Use of REMIND Artificial Intelligence Software for Rapid Assessment of Adherence to Disease Specific Management Guidelines in Acute Coronary Syndromes Ali F. Sonel, MD, C. Bernie Good, MD MPH, Harsha Rao, MD, Alanna Macioce, BS, Lauren J. Wall, BS, Radu Stefan Niculescu, PhD, Sahtyakama Sandilya, PhD, Phan Giang, PhD, Sriram Krishnan, PhD, Prasad Aloni, MS, MBA, Bharat Rao, PhD Center for Health Equity Research and Promotion, VA Pittsburgh Healthcare System and the Cardiovascular Institute, University of Pittsburgh Pittsburgh, PA, Siemens Medical Solutions, USA, Malvern, PA ABSTRACT Introduction: Manual extraction of data for Quality Improvement is tedious, requiring significant individual training and careful attention to the HIPAA Privacy Rule. Automated chart abstraction is an alternative approach that saves time and costs. We compared manual chart abstraction from an electronic medical record (VA CPRS EMR System) to automated extraction using the REMIND artificial intelligence software in 327 consecutive patients admitted with unstable angina or non-ST elevation myocardial infarction. Methods: All patient features required by ACC/AHA guidelines for determining eligibility for class I recommendations to use aspirin, beta-blockers, heparin, glycoprotein IIb/IIIa receptor antagonists, and ACE inhibitors were extracted by both methods. Manual extraction was carried out by well-trained, qualified chart abstractors with prior experience in manual chart abstraction. When both extraction results were identical, the result was assumed correct. Disagreements were manually adjudicated based on pre-determined definitions. Results: Manual extraction and data entry required 176 hours compared to 4 hours using the Siemens REMIND software. A total of 5232 data elements were identified, with agreement in 4385 (84%) and disagreement in 847 (16%), involving % of patients for various parameters. REMIND was found to be correct in 642 disagreements (76%) and manual extraction was correct in the remaining 24% (205). Based on adjudication, REMIND identified adherence compared very favorably to manual extracted as well as adjudicated guideline adherence for aspirin (83% vs. 88% vs. 85%), beta blockers (78% vs.82% vs. 81%), heparin (53% vs. 51% vs. 54%), glycoprotein IIb/IIIa receptor antagonists (35% vs. 38% vs. 40%) and ACE inhibitors (69% vs. 78% vs. 76%). Conclusions: REMIND can assess disease specific management guideline adherence at least as accurately as manual chart abstraction. Use of REMIND for Quality Improvement and research can result in significant savings, better resource utilization, and may improve data extraction quality. BACKGROUND Research and quality improvement projects involve large amounts of data collection through review of medical records Manual data collection requires a significant amount of training and is time consuming Automated data extraction methods could save time and improve resource utilization Little is known about the accuracy of automated systems for record extraction METHODS Patient Population 327 patients admitted with high-risk non-ST-segment elevation myocardial infarction were included in the study Data Collection Records were extracted from VA CPRS Electronic Medical Record System Manual extraction of predefined variables was performed by a trained abstractor with expertise in ACS data abstraction for research purposes An artificial intelligence model developed by Siemens, the REMIND automated data extraction tool, was used to extract the same information electronically Medical information required to determine eligibility and the presence of absence of contraindications for Class I treatment recommendations in the ACC/AHA guidelines was collected for the following medications: Aspirin in all patients Beta-blockers in all patients Heparin in all patients Angiotensin converting enzyme (ACE) inhibitors or angiotensin receptor blockers (ARB) in patients with diabetes mellitus, congestive heart failure, left ventricular dysfunction or hypertension Glycoprotein IIb/IIIa receptor antagonists in patients in whom an early invasive management strategy is planned Data Analysis We compared the results of the two methods for accuracy When both extraction methods were in agreement, the result was assumed to be correct. When extracted results differed, disagreements were manually adjudicated based on pre-determined definitions, using the source documents of each extraction method Accuracy was defined as the number of patients where there was agreement with adjudication as to whether the patient was compliant or not, divided by the total number of patients in the study Compliance is defined as the number of patients eligible and not contraindicated to that medication, who actually received the medication, divided by the number of patients who are eligible and have no contraindication to that medication. RESULTS Complete data extraction required 176 hours of manual extraction, compared to 4.5 hours with REMIND automated extraction TREATMENT ACCURACY (%) N=327 REMINDMANUAL Aspirin319 (98%)314 (96%) Beta Blockers319 (98%)316 (97%) Heparin315 (96%)296 (91%) ACE Inhibitors/ARB300 (92%)310 (95%) Glycoprotein IIb/IIIa Receptor Antagonists 300 (92%)290 (89%) CONCLUSION Use of REMIND for quality improvement and research related applications in facilities with electronic medical records can result in significant savings and better resource utilization. Use of REMIND can enable evaluation of very large sets of medical information that would otherwise be impractical by manual extraction TREATMENT COMPLIANCE BY REMIND EXTRACTION (%) COMPLIANCE BY MANUAL EXTRACTION (%) COMPLIANCE FOLLOWING ADJUDICATION (%) Aspirin202/245 (83%)220/250 (88%)206/241 (85%) Beta Blockers76/98 (78%)116/141 (82%)76/94 (81%) Heparin131/245 (53%)127/250 (51%)131/241 (54%) ACE Inhibitors/ARB101/146 (69%)98/125 (78%)95/125 (76%) Glycoprotein IIb/IIIa Receptor Antagonists 36/103 (35%)30/79 (38%)34/86 (40%) REMIND can determine ACC/AHA guideline adherence for non-ST-elevation acute coronary syndromes at least as accurately as manual chart abstraction. IMPLICATIONS TREATMENT (N=327) Patients with Contraindications for Processes of Care (%) Ideal Patients for Processes of Care (%) REMINDMANUALADJUDICATED REMINDMANUALADJUDICATED Aspirin82 (25%)77 (24%)86 (26%)245 (75%)250 (77%)241 (74%) Beta Blockers229 (70%)186 (57%)233 (71%)98 (30%)141 (43%)94 (29%) Heparin82 (25%)77 (24%)86 (26%)245 (75%)250 (77%)241 (74%) ACE Inhibitors/ARB 142 (43%)128 (39%)152 (46%)146 (45%)125 (38%) Glycoprotein IIb/IIIa Receptor Antagonists 99 (30%)93 (28%)106 (32%)103 (32%)79 (24%)86 (26%) Table 2: Assessment of Compliance with Guideline Recommended Therapies Table 3: Accuracy* of Compliance Assessment with REMIND Compared to Manual Extraction Table 1: Determination of Contraindications and Eligible Patients for Processes of Care *Accuracy defined as true positives plus true negatives divided by the total number of patients SPECIFIC AIMS Compare the accuracy of data collection in a large and complex medical record set using manual extraction and REMIND automated extraction tool Compare the level of adherence to ACC/AHA guideline recommendations for treatment of non-ST elevation acute coronary syndromes (ACS) using manual extraction and REMIND automated data extraction tool