1. Using text mining methods to detect a clinical infection
Milena Gianfrancesco, PhD MPH
Postdoctoral Researcher
Division of Rheumatology
UCSF School of Medicine
Suzanne Tamang, PhD
Assistant Faculty Director, Data Science
Stanford Center for Population Health Sciences
05/18/2018
4/3/2019
[ADD PRESENTATION TITLE: INSERT TAB > HEADER & FOOTER > NOTES AND HANDOUTS]

2. Zoster infection a.k.a. “shingles”
Reactivation of the virus that causes chickenpox: varicella zoster virus
1 out of every 3 people will develop zoster in their lifetime.
Anyone with a history of chickenpox can get zoster, but the risk generally increases with age.
Patients on an immunosuppressive drug have a higher risk of developing zoster, and it can be more severe.
Limited knowledge available to determine which patients are at highest risk for zoster, information critical for implementing preventive strategies such as vaccination or antiviral prophylaxis.
*If we knew that a certain medication was associated with infection as well as a person’s characteristics, could vaccinate those first or consider switching medications.
More severe cases: hospitalizations, central nervous system manifestations with potentially fatal or long term disabling outcome can occur.
Lastly, new vaccine (killed) that’s more effective (vs live vaccine currently); can allocate this to those at high risk.
[(4/1000 U.S. annually, age adjusted; above 60 years, 10 per 1,000 U.S. population)]
Performance of machine learning methods using electronic medical records to detect and predict a clinical infection
4/3/2019
4/3/2019
[ADD PRESENTATION TITLE: INSERT TAB > HEADER & FOOTER > NOTES AND HANDOUTS]

3. Clinical reporting of zoster
Zoster infection is often treated outside of specialty clinic
Sometimes entered as diagnosis (i.e. ICD code); more often mentioned in clinical note
ICD codes may underestimate prevalence
Potential bias towards more severe cases
VZV often treated outside of specialty clinic, in outpatient setting or at an outside PCP
Performance of machine learning methods using electronic medical records to detect and predict a clinical infection
4/3/2019
4/3/2019
[ADD PRESENTATION TITLE: INSERT TAB > HEADER & FOOTER > NOTES AND HANDOUTS]

4. Goals of project
Apply and validate a text mining system to extract incident zoster infection from clinical notes
Do ICD codes truly associate with more severe cases of zoster?
Performance of machine learning methods using electronic medical records to detect and predict a clinical infection
4/3/2019
4/3/2019
[ADD PRESENTATION TITLE: INSERT TAB > HEADER & FOOTER > NOTES AND HANDOUTS]

5. Study population UCSF EHR
Data from June 1, 2012 – November 5, 2016 for 800,000+ individuals
Structured tables
Unstructured data (e.g. clinical notes)
Performance of machine learning methods using electronic medical records to detect and predict a clinical infection
4/3/2019
4/3/2019
[ADD PRESENTATION TITLE: INSERT TAB > HEADER & FOOTER > NOTES AND HANDOUTS]

6. Study sample
Individuals prescribed an immunosuppressant medication, and > 2 encounters in EHR 30 days apart.
31 immunosuppressant medications (IM) included
N= 36,042 IM orders
N= 16,344 unique individuals
N= 259 cases identified via ICD code
EHR
N~800,000
Zoster prevalence in general population ~ 0.5% IM
N~16,000
Zoster prevalence in
IM population ~ 1.6%
Performance of machine learning methods using electronic medical records to detect and predict a clinical infection
4/3/2019
4/3/2019
[ADD PRESENTATION TITLE: INSERT TAB > HEADER & FOOTER > NOTES AND HANDOUTS]

7. Demographics of participants (n=16,344)
N (%) or
Mean (SD)
Female
 8,506 (52%)
Age
50.31 (19.60)
Race
White
Asian
Black
Other
Unknown/Declined
8,304 (51%)
2,206 (13%)
1,111 (7%)
3,810 (23%)
859 (5%)
Ethnicity
Non-Hispanic or Latino
Hispanic or Latino
12,589 (77%)
2,867 (18%)
888 (5%)
Diverse sample but a majority were non Hispanic white
43% of sample was under 50 years old; traditionally zoster associated with age > 50 and currently recommended to vaccinate that age group; no recommendations for those under
61% under 60 years old.
Performance of machine learning methods using electronic medical records to detect and predict a clinical infection
4/3/2019
4/3/2019
[ADD PRESENTATION TITLE: INSERT TAB > HEADER & FOOTER > NOTES AND HANDOUTS]

8. [CL]inical [EVE]nt [R]ecognition
Tricks based a less formal representation of text, more aligned with informal text, and aimed at efficiency, domain knowledge, common sense, and neat little tricks. CLEVER rules.

9. Challenges of Clinical Text Analysis
Clinical notes are not SOAP notes
EMR > text (IE, not NLP!)
Boundary detection is challenging (context window)
End to end? Maybe not…
CLEVER: assume there are important local contexts
Synonyms, lexical variants (seed terms)
Highly ambiguous (task specific lexicon)
Semantic modifiers (base classes)
Subgrammer, sublanguage (word embeddings)
Acronyms
Colloquial terms
Out of UMLS vocabulary

10. CLEVER Pipeline 1. Terminology Construction 4. Patient-level Reporting
Candidate Event Matrix p0 t0
PID
Event p0 … 1 pn
2. Pre-processing
Structured
t364
Unstructured
PID
Event p0 … 1 pn
______
Events
Unstructured EHR Data = pn … … … … … … … …
Eligible Patients … … … … … … … …
target sequence
class sequence
candidate id
note section
time offset
target term
patient id
note type
...
Clinical Text … … … … … … … …
Patient Labels
time offset … … … … … … … … … … … … … … … …
Combined Events … … … … … … … …
time offset
CPT codes
ICD codes
patient id
gender
age …
Tokenizer
3. Extraction
Structured Encounter Data
cid0
Section Detector
cid1
Event-level Labels
cid2
Class Sequencer p0
N-gram Ranker p1 … … … … … … … … …
cidm
Rule-based Extractor
Eligible Patients p2
Concept Recognizer*
Statistical Extractor pn … … … … … … … …
Qualifying Criteria
* We do not use a distinct concept extraction step this work, but files for the purpose are produced by CLEVER

11. Example of zoster dictionary
Explain classes(?); i.e. VCV; VCV rx; VCV low
4/3/2019
[ADD PRESENTATION TITLE: INSERT TAB > HEADER & FOOTER > NOTES AND HANDOUTS]

12. Labeled Output: “Negative”
SNIPPET: . He was admitted [DATE] for evaluation and management of likely varicella zoster infection. His symptoms began as L-sided mouth pain ~6-7 days PTA which has become progressively worse. He was initially seen at the…
CLEVER ANNOTATION: NEGATIVE|SCREEN|DOT_SCREEN_#VCV#_DOT_DOT|
zoster infection|PID|NID|Consults|DATETIME|7|VCV|807|1059 |UK|NULL|period:DOT:1:986:73,evaluation:SCREEN:686:1013:46, period:DOT:1:1075:16,period:DOT:1:1168:109
Example 1: “negative” / “risk” = tagging as negative because in past (“she had.. Infection”)
Example 2: two snippets from same person but see that each mention is tagged as it’s own. From this we’d have to decide for each person how to label (again, # positive mentions / all mentions?)

13. Labeled Output: Positive
SNIPPET: Would continue supportive care and refrain from using nephrotoxic agents at this time until pt demonstrates renal recovery. Zoster in immunocompormised: would recommend decrease dose of Acyclovir to 350mg Q8 and treat for the minimal treatment time.
CLEVER ANNOTATION:
POSITIVE|VCV|PT_DOT_#VCV#_PUNCT_DOT|zoster|PID|NID|Ambulatory Progress Notes|DATETIME|3|VCV|745|3960|ct| 3326|pt:PT:8:3928:32,period:DOT:1:3958:2,colon:PUNCT:4:3987:27, period:DOT:1:4084:124
Example 1: “negative” / “risk” = tagging as negative because in past (“she had.. Infection”)
Example 2: two snippets from same person but see that each mention is tagged as it’s own. From this we’d have to decide for each person how to label (again, # positive mentions / all mentions?)

14. Zoster case detection using CLEVER
Generated a ‘dictionary’ of terms associated with zoster to assist in labeling notes
Ran CLEVER on all notes
Compiled files for each patient
All positive mentions
All negative mentions
Will join with structured data (ICD codes, meds, labs, age, sex, race, etc.) to help identify case status
Need to determine heuristic (i.e. # positive mentions / all mentions) to guide in labeling as “case”

15. Conclusion
Further refinement of CLEVER to detect all types of infections will assist in developing a highly accurate pipeline for adverse event detection
Better phenotyping of outcomes will assist future studies in identifying risk factors to prevent occurrences of adverse events
Performance of machine learning methods using electronic medical records to detect and predict a clinical infection
4/3/2019
4/3/2019
[ADD PRESENTATION TITLE: INSERT TAB > HEADER & FOOTER > NOTES AND HANDOUTS]

16. Acknowledgements Funding: AHRQ: R01 HS024412 (PI: Yazdany)
NIAMS: F32 AR (PI: Gianfrancesco)
Rheumatology Quality and Informatics Laboratory (QUIL)
Jinoos Yazdany
Gabriela Schmajuk
Dana Ludwig
Steve Shiboski
Laura Trupin
Michael Evans
Julia Kay
Zara Izadi
Jing Li
Performance of machine learning methods using electronic medical records to detect and predict a clinical infection
4/3/2019
4/3/2019
[ADD PRESENTATION TITLE: INSERT TAB > HEADER & FOOTER > NOTES AND HANDOUTS]