Evaluation of the metrological quality of medico-administrative data for perinatal indicators: a pilot study in three French university hospitals K.Goueslard1 A.Pierron3 M.Revert1 A. Vuagnat2 J.Cottenet1 E. Benzenine1 J. Fresson3 C. Quantin1,4 Thank you for giving me the opportunity to present our study about the metrological quality of medico-administrative data for perinatal indicators. This study was coordinated by Pr C. Quantin. 1.Department of medical information-Dijon, France 2.DREES department of research and evaluation – Ministry of Health 3.Department of medical information-Nancy 4.Inserm UMR 1181 « Biostatistics, Biomathematics, Pharmacoepidemiology and Infectious Diseases » 1

Background Exhaustive and standardised information for Indicators EUROPERISTAT Public health studies Medico-administrative database 800,000 deliveries/year Comparison with National Perinatal Enquiry In perinatal health, exhaustive and standardized information is required to compare indicators with those in other European countries and for public health studies. The Medico-administrative database has particular interest in France since almost all 800 000 births take place in hospital. We have already compared aggregated statistics from the medico- administrative database with National Perinatal Enquiry aggregated data and shown the interest of using such data, as the results are very close. We would like now to undertake epidemiological studies using hospital data and need to have more information about the quality of individual data. 2

Pilot study Assess the metrological quality of medico-administrative data (discharge abstracts) for core indicators in three university hospitals Compare the medico-administrative data to the medical record considered as the Gold Standard The aim of our study was to assess the metrological quality of the medico-administrative data for core indicators in three university hospitals. We propose to compare hospital data with medical records which we consider the gold standard. 3

Selection of discharge abstracts Three university hospitals: Paris Port-Royal, Nancy, Dijon Discharge abstracts (2012) Selection of deliveries from French Common Classification of Procedures Random selection of 100 live births per hospital Nancy Paris Dijon This pilot study took place in 3 university hospitals of a large, a middle and a small university town. Selection of deliveries was conducted using procedures coded in discharge abstracts, according to the French Common Classification. We developed a specific program to select at random one hundred live births by hospital among the eleven thousand fifteen eligible stays in these hospitals during the year two thousand and twelve. 4

Collection from medical records Characteristics collected by midwives Stay (length of stay, …) Women (age, obesity, parity, …) Pregnancy (diabetes, hypertensive disorders, …) Delivery (mode, epidural, postpartum haemorrhage, …) Children born (gender, weight) In medical records, data were collected by midwifes from a standardized collection grid, which included characteristics of the hospital stay, the women, the pregnancy, the delivery and the children born. 5

Comparison discharge abstracts/medical records Positive Predictive Value (PPV) Sensitivity Concordance rate Medical record Medico Adinistrative Data + - PPV FP FN NPV PPV: Positive Predictive Value FP: False Positive NPV: Negative Predictive Value FN: False Negative The positive predictive value and the sensitivity were calculated for dichotomous data. Continuous data were assessed by the concordance rate. PPV: probability that the information in the discharge abstract was also present in the medical record. Se: probability that the information in the medical record was also present in the discharge abstract. Concordance rate: number of concordant cases between discharge abstracts and medical records. 6

Maternal characteristics Medical record Medico administrative data PPV [CI 95%] Sensitivity Age < 20 7 100 20-24 28 29 93.1 [83.9-100] 96.4 25-29 88 91 94.4 [89.6-92.2] 96.6 30-34 104 101 99.0 [97.1-100] 96.2 35-39 52 ≥ 40 20 DM 1 Parity Primiparous 109 116 91.4 [86.3-96.5] 97.3 Multiparous 126 118 96.6 [93.3-99.9] 90.5 2 As regards the average maternel age, the concordance rate was 93%. If we consider age classes, we can also compute the positive predictive value and the sensitivity which were within between 93 and 100%. In 2012, parity was only available for vaginal deliveries. The PPV was ninety-one point four (91.4) per cent for primiparous women and 96.6 per cent for multiparous women. 7

Medico administrative data Maternal morbidity Medical record Medico administrative data PPV [CI 95%] Sensitivity Diabetes 27 20 100 74.1 Pre-existing Diabetes 2 Gestational Diabetes 22 18 88.9[74.3-100] 72.7 Diabetes, not specified 3 - Obesity 33 32 81.3 [67.8-94.8] 78.8 Hypertensive disorders 19 63.0 Gestational Hypertension 4 66.7 [13.4-100] 50.0 Pre-eclampsia HELLP Syndrome 16 13 81.3 Postpartum Haemorrhage 41 38 90 [79.8-99.2] 82.9 The results for maternal morbidity showed that the positive predictive value was 100% and the sensitivity was 74.1% for diabetes. The positive predictive value was only 88.9% for gestational diabetes. In thirty-two hospital stays, the positive predictive value for obesity was 81.3%. We studied hypertension disorders during pregnancy and found a positive predictive value of 100%. The positive predictive value for pre-eclampsia and HELLP syndrome was also 100% but the Se was only 81.3%. Lastly, the positive predictive value for postpartum hemorrhage was 90%. 8

Delivery Characteristics Medical record Medico administrative data PPV [CI 95%] Sensitivity Gestationnal age <37 33 93.9 [85.8-100] 74.1 Simple birth 289 100 Twins birth 11 Epidural 211 209 96.2 [93.6-98.8] 93.3 Vaginal delivery 197 198 99.5 [98.8-100] Instrumental extraction 40 39 97.5 Caesarean 62 63 98.4 [95.3-100] Caesarean in emergency 48 51 92.2 [84.8-96.6] 97.9 Cephalic presentation 195 193 99 Breech presentation 3 5 60.0 [17.1-100] Concerning the delivery characteristics, the positive predictive value and sensitivity for simple and twin births were one hundred per cent. As regards the mode of delivery, the positive predictive value was 99.5% for vaginal delivery, 100% for instrumental delivery, 98.4 for caesarean and 92.2 when the caesarean took place in an emergency. The positive predictive value was 100% and the sensitivity was 99% for cephalic presentation among the 193 hospitals stays that provided data on this. For some variables like gestational diabetes and obesity, the positive predictive value varied considerably between hospitals: PPV varied from 74,3% to 100% for gestational diabetes and from 67,8% to 94,8% for obesity. 9

Discussion Aggregated level: frequency of pathologies Aggregate data: false negatives & false positives Aggregated level: frequency of pathologies Individual level: longitudinal studies Professional practices: differences in coding practises Postpartum Haemorrhage Medical record: 41 Hospital data: 38 We consider that this type of study to assess individual data is essential as aggregate data don’t assess the impact of false positives and false negatives which may balance each other out. For example, postpartum hemorrhage presented similar numbers in the two databases but only a positive predictive value of 89,5%. As a consequence, we think that, when the validation is performed at an aggregated level, medico- administrative data can be used to estimate the frequency of pathologies. However, for longitudinal studies, it is necessary to assess individual data. It is also important to be careful when studying professional practices from hospital databases as the inter-hospital differences may only reflect differences in coding practices. Our results differed of course from one indicator to another. The indicators based on consensus definition and having an influence on the hospital budget (gestational age, simple or twin births, mode of delivery, presentation) were of course better collected. Indicators with less standardized definitions were less documented. This was the case for pathologies for which the formulation of wordings in the medico-administrative database may differ from the terms used in clinical practice. Specific training and validation seem to be needed to achieve the objectives of hospital financing, but also to meet the ever-increasing use of medico-administrative databases for epidemiological purposes. 10

Discussion Consensual definition: well-collected Difference between indicators Consensual definition: well-collected Pathologies: less documented Specific training and validation

Conclusion Medico-administrative data more reliable for two reasons ? Importance for budgetary promotion in hospitals Increasing use for statistical and epidemiological purposes National study: 25 hospitals In conclusion, since 2012 in France, medico-administrative data have become more reliable for two reasons: - The importance of this data for budgetary purposes in hospitals - The increasing use of this information for statistical and epidemiological purposes by research and government institutions. This study produced preliminary results which should be confirmed by a national study which is financed by the National Agency for Research and will begin soon. It will include twenty-five hospitals and major indicators of perinatal health, such as prematurity and perinatal mortality. 12

13

Medical record Administrative information Medical history Pregnancy monitoring Delivery report +/- operative report Discharge letter

PPV: Probability that the information in discharge abstract was present in medical record Sensitivity: Probability that the information in medical record was present in discharge abstract Concordance rate False positive (Type I error): Presence in hospital data since absence in medical record False negative (Type II error): Absence in hospital data since presence in medical record

Selection of delivery Z37 may: Be identified in postpartum stay Take place in establishment other than delivery place French common classification of procedures for delivery All types of deliveries Whatever presentation Spontaneous vaginal deliveries Instrumental vaginal deliveries Asisted vaginal deliveries Caesarean Simple or multiple births

Maternal characteristics PPV [CI 95%] Sensitivity [CI 95%] Age < 20 100 20-24 93.1 [83.9-100] 96.4 [89.5-100] 25-29 94.4 [89.6-92.2] 96.6 [92.6-100] 30-34 99.0 [97.1-100] 96.2 [92.4-100] 35-39 ≥ 40 DM Parity Primiparous 91.4 [86.3-96.5] 97.3 [94.2-100] Multiparous 96.6 [93.3-99.9] 90.5 [85.4-95.6] As regards the average maternel age, the concordance rate was 93%. If we consider age classes, we can also compute the positive predictive value and the sensitivity which were within between 93 and 100%. In 2012, parity was only available for vaginal deliveries. The PPV was ninety-one point four (91.4) per cent for primiparous women and 96.6 per cent for multiparous women. 17

Maternal morbidity Diabetes Obesity Hypertensive disorders PPV [CI 95%] Sensitivity [CI 95%] Diabetes 100 74.1 [57.8-90.6] Pre-existing Diabetes Gestational Diabetes 88.9[74.3-100] 72.7 [54.1-91.3] Diabetes, not specified - Obesity 81.3 [67.8-94.8] 78.8 [64.9-92.7] Hypertensive disorders 63.0 [44.8-81.2] Gestational Hypertension 66.7 [13.4-100] 50.0 [1.0-99.0] Pre-eclampsia HELLP Syndrome 81.3 [62.2-100] Postpartum Haemorrhage 90 [79.8-99.2] 82.9 [71.4-94.4] The results for maternal morbidity showed that the positive predictive value was 100% and the sensitivity was 74.1% for diabetes. The positive predictive value was only 88.9% for gestational diabetes. In thirty-two hospital stays, the positive predictive value for obesity was 81.3%. We studied hypertension disorders during pregnancy and found a positive predictive value of 100%. The positive predictive value for pre-eclampsia and HELLP syndrome was also 100% but the Se was only 81.3%. Lastly, the positive predictive value for postpartum hemorrhage was 90%. 18

Delivery Characteristics PPV [CI 95%] Sensitivity [CI 95%] Gestationnal age <37 93.9 [85.8-100] 74.1 [85.7-100] Simple birth 100 Twins birth Epidural 96.2 [93.6-98.8] 93.3 Vaginal delivery 99.5 [98.8-100] Instrumental extraction 97.5 [92.6-100] Caesarean 98.4 [95.3-100] Caesarean in emergency 92.2 [84.8-96.6] 97.9 [93.8-100] Cephalic presentation 99 [97.6-100] Breech presentation 60.0 [17.1-100] Concerning the delivery characteristics, the positive predictive value and sensitivity for simple and twin births were one hundred per cent. As regards the mode of delivery, the positive predictive value was 99.5% for vaginal delivery, 100% for instrumental delivery, 98.4 for caesarean and 92.2 when the caesarean took place in an emergency. The positive predictive value was 100% and the sensitivity was 99% for cephalic presentation among the 193 hospitals stays that provided data on this. For some variables like gestational diabetes and obesity, the positive predictive value varied considerably between hospitals: PPV varied from 74,3% to 100% for gestational diabetes and from 67,8% to 94,8% for obesity. 19