1. Automatic Formalization of Clinical Practice Guidelines Matthew S. Gerber and Donald E. Brown Department of Systems and Information Engineering University of Virginia James H. Harrison Department of Public Health Sciences University of Virginia

2. Many treatment options – what to do? Clinical Practice Guidelines 2 Benefits / costs Evidence quality Randomized clinical trial: beneficial Expert opinion: might be beneficial Meta-analysis: usually beneficial Strength Recommended Should consider Might consider

3. Clinical Practice Guidelines Development –Expert synthesis of current evidence –Example from heart failure: 3

4. Clinical Practice Guidelines Expected outcomes –Evidence-based clinical decision aid –Reduction in cost and treatment/outcome variation –Improvement in patient health Challenges –A guideline for any occasion –Guidelines change periodically –Lengthy (HFSA CPG is 259 pages) 4 Total (NGC)2,269 Cardiovascular diseases486 Heart failure152

5. Clinical Decision Support Systems Goal: deliver CPG knowledge at point of care Alleviate burden on clinician Problem: CPGs contain minimally structured text 5 Formalization is required

6. Traditional CPG Formalization 6 Knowledge engineers Medical experts Knowledge representation Knowledge management software (e.g., Protégé) CDSS CPG Automatic formalization

7. … Retrospective analyses The Big Picture 7 Structured knowledge Medical decision support ? Endocrine Infections … Cardiovascular NLP

8. Data Collection Yale Guideline Recommendation Corpus –Hussain et al. (2009) –1,275 recommendations –Representative sample of domains and rec. types “Oral antiviral drugs are indicated within 5 days of the start of the episode and while new lesions are still forming.” –Simplifications Delimited recommendations No inter-recommendation dependencies Random sub-sample of YGRC (n=200) 8

9. Recommendation Representation SNOMED-CT –Medical concept ontology –Broad coverage 9 Keywords ? Asbru, etc. Fidelity: Low High Automation: Trivial Impossible

10. Recommendation Representation 10 (Sundvalls et al., 2012)

11. Recommendation Representation 11 PrimarySecondaryExample recommendation ACTIONEVALUATION[computed tomography CT] should be used PROCEDURE[red blood cell transfusion] is appropriate … EVIDENCESTRONG[it has been shown to reduce the occurrence of NTDs] WEAK/NONE[there is insufficient evidence] MODALITYOBLIGATORYcomputed tomography CT [should] be used NEVERoral risedronate [should not] be used OPTIONALphysician [may] choose AGENT[physician] may choose MORBIDITYprevent [preeclampsia] POPULATION[obese women with gestational diabetes mellitus] PURPOSEis used [to prevent osteoporotic fractures] TEMPORAL[initial] treatment SNOMED-CT CONCEPT: 129265001

12. Recommendation Annotation Task: manually identify representational elements within recommendations Example Diuretics are recommended for patients with heart failure. [ DRUG Diuretics] are recommended for [ POPULATION patients with [ MORBIDITY heart failure]]. 12

13. Methods Natural language processing Supervised classification Per-recommendation pipeline 1.Syntactic parsing 2.Parse node classification 3.Post-processing 13

14. Methods: (1) Syntactic Parsing 14 Constituency parser (Charniak and Johnson, 2005)

15. Methods: (2) Parse Node Classification 15 Unit of classification: node Multi-class logistic regression Example: 1 positive, 17 negative Actual –12K nodes –10 classes (primary)

16. Methods: (2) Parse Node Classification 16 Linguistic features –Word stems under node –Syntactic configuration of node –…

17. Methods: (2) Parse Node Classification 17 Learning –Forward feature selection –Per-class costs (LibLinear)

18. Methods: (3) Post-processing 18 Remove duplicates Other possible issues –Conflicts –Embedding

19. Evaluation Results 19 Element#Precision (%)Recall (%)F1 (%) ACTION30174.526.238.7 MODALITY15871.973.072.4 POPULATION14083.756.667.5 TEMPORAL5328.61.12.2 TRIGGER4581.793.387.1 PURPOSE4358.316.325.5 EVIDENCE3883.313.222.7 AGENT3776.047.658.5 MORBIDITY1950.010.517.4 Overall83475.341.753.7 10-fold cross-validation

20. Discussion High variance across classes Alternative strategies –Identify more informative features –Change the model formulation –Annotate more data 20

21. Conclusions CPGs are an important knowledge source Difficult to use within CDSS Prior CPG formalization –Manual –Automatic for specific domains / recommendations Our contributions –SNOMED-CT representation –Manually annotated recommendation sample –Statistical NLP model / evaluation 21

22. Future Work Refined representation Model formulation Feature engineering Controlled natural language 22

23. Questions? References 1.Charniak, E. & Johnson, M. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, 2005, 173-180. 2.Hussain, T.; Michel, G. & Shiffman, R. N. The Yale Guideline Recommendation Corpus: A representative sample of the knowledge content of guidelines. I. J. Medical Informatics, 2009, 78, 354-363. 3.Fan, R.-E.; Chang, K.-W.; Hsieh, C.-J.; Wang, X.-R. & Lin, C.-J. LIBLINEAR: A Library for Large Linear Classification. Journal of Machine Learning Research, 2008, 9, 1871-1874. 4.Sundvall, E.; Nystrom, M.; Petersson, H. & Ahlfeldt, H. Interactive visualization and navigation of complex terminology systems, exemplified by SNOMED CT. Studies in health technology and informatics, IOS Press; 1999, 2006, 124, 851. 23