1. A Scalable Machine Learning Approach for Inferring Probabilistic US-LI-RADS Categorization
NLP and Machine Learning
S54
Imon Banerjee, Hailey H. Choi,
Terry Desser, Daniel L. Rubin
Biomedical Data Science, Stanford University
Twitter: #AMIA2018

2. Disclosure
I and my spouse/partner have no relevant relationships with commercial interests to disclose.

3. Motivation
Liver ultrasound is recommended for screening and surveillance of patients at high-risk for hepatocellular carcinoma (HCC)
Radiology department performs thousands of screening ultrasounds, but we rarely receive feedback on our effectiveness. Sought to develop a tool to rapidly review the results of tens of thousands of ultrasound exams over more than one decade.
In 2017, American College of Radiology (ACR) introduced Ultrasound Liver Imaging Reporting and Data System (LI-RADS) for standardization of reporting
How to longitudinally follow-up patients before and after LI-RADS reporting?
How to track patients across multiple institutions?

4. Challenge: US Reports with and without template
LI-RADS formatted report
NON-LI-RADS formatted report
FINDINGS:
Liver: Visualization: A--No or minimal limitations in liver visualization.
Liver length: 11.9 cm.
Liver appearance: Moderately cirrhotic.
Liver observations: The right hepatic lobe in segment 7/8 shows a rim calcified, thinly septated hypoechoic lesion with posterior through transmission likely representing a partially calcified cyst measuring 1.2 x 1.5 x 1.3 cm. No internal vascularity or nodularity identified within this cystic structure. Otherwise, the liver shows no suspicious hepatic lesion.
Liver Doppler: Hepatic veins: Patent with normal triphasic waveforms.
Portal veins: Patent with normal hepatopetal flow.
Portal vein velocity: 45.7 cm/sec.
Superior mesenteric and splenic veins: Patent with normal hepatopetal flow.
Hepatic artery: Patent with a normal waveform.
Hepatic artery velocity: cm/sec.
Hepatic artery resistive index: 0.61.
IMPRESSION:
1.US LI-RADS 1--Negative. Recommend routine follow-up surveillance ultrasound in 6 months.
2.Visualization score: B--Moderate limitations that may obscure small masses. Limitations are due to cirrhosis.
3.Nonspecific elevation of the hepatic artery velocity which may be seen in hepatic inflammation.
4.Hepatic cirrhosis with increased splenomegaly and new ascites.
Multiple sonographic images of the abdominal structures are performed. Innumerable hyperechoic and hypoechoic solid nodules, and cystic lesions are seen within the liver. Direct comparison with the previous exam is essentially not possible given the multiplicity of lesions. The spleen is normal in echo-texture and size, with a maximum dimension of 9.9 cm. Both kidneys are normal in echogenicity and size without evidence for hydronephrosis or nephrolithiasis. The right kidney measures 10.5 cm in length, and the left kidney measures 9.8 cm in length. The gallbladder is unremarkable without evidence of wall thickening, stones, or pericholecystic fluid collections. The main portal vein is patent and exhibits normal hepatopetal flow.
innumerable lesions are identified within the liver. On the prior CT, dated XX, THESE included a hypervascular focus. Follow-up imaging in this patient should be performed with multiphasic CT or MRi as was recommended at that time.
END OF IMPRESSION:
SUMMARY: 4:-POSSIBLE SIGNIFICANT FINDINGS, MAY NEED ACTION
Not trivial to infer the LI-RADS score
Easy to extract LI-RADS score

5. ACR LIRADS Scoring Surveillance US exam in high-risk patient
No observations or definitely benign observations only
One or more observations, not definitely benign
New thrombus
in vein
Diameter
< 10 mm
≥ 10 mm
LI-RADS 1
Negative
LI-RADS 2
Subthreshold
LI-RADS 3
Positive

6. f( )= LI-RADS ? Objective
Propose an automated method to assign LI-RADS categories to narrative radiology reports to facilitate such follow up.
Facilitate deep learning image-based research with US images by offering large scale text mining and data gathering opportunities from standard hospital clinical data repositories. f(
)= LI-RADS ?

7. Liver Ultrasound Cohorts
Stanford Data
Used for testing
Year (without LI-RADS template) -
11,154
Year 2017 (without LI-RADS template)
962
Used for training and validation
Year 2017 (with LI-RADS template)
Total
LI-RADS 1
1589
LI-RADS 2 93
LI-RADS 3 62
UT Southwestern Data
Used for testing
Year 2017 (with LI-RADS template)
Total
LI-RADS 1
1867
LI-RADS 2
162
LI-RADS 3
118

8. Distributed semantic pipeline
Learning LI-RADS coding
Reports after 2017
With LI-RADS?
[ ]
Trained classifier
Learning word semantics
LIRADS
vocabulary
Reports before 2017
Infer LI-RADS coding
Validated on non-LIRADS reports
Validated on different institution
Annotated US reports of Stanford
[2007 – 2016]
Annotated US reports UT Southwestern

9. Learnt word semantics Synonyms of LI-RADS terminology Categories
LI-RADS Lexicon
Synonyms generated
Echogenicity
hyperechoic
hyperechogenic, hyperecho
isoechoic
isoecho
hypoechoic
hypoechogenicity, hypoechogen, hypoecho
cystic
anecho, anechoic
nonshadowing
non_shadowing
Doppler vascularity
hypovascular
nonenhancing
avascular
nonvascular
hypervascular
hypervascularity
Architecture
septation
septat, septations, multicystic, septa, complex_cyst, intern_septation, thin_septation, multispet, reticul, fishnet, multilocul
complex
complicated, solid_and_cystic
Morphology
lobulated
bilobe, macrolobulated, microlobulated
round
oval, rounded, ovoid, oblong
ill-define
vague, indistinct
exophytic
bulge
well_defined
well_circumcribed, marginated

10. Ensemble classifier Lesions - 2 Long axis – 18 mm
FINDINGS: Pancreas: Visualized portions normal. Liver length: 12.8 cm. Liver appearance: Redemonstration of cirrhotic morphology. There is a 1.1 x 1.4 x 1.1 cm hypoechoic mass in the anterior left hepatic lobe with irregular margins and internal vascularity. Additional hypoechoic mass measuring 1.3 x 1.7 x 1.8 cm is visualized in the upper right hepatic lobe. No definite internal vascularity is seen within the right hepatic lobe mass. No biliary ductal dilatation. Hepatic veins: Patent with normal triphasic waveforms.
Portal veins: Patent with normal hepatopetal flow.
Portal vein velocity: 24 cm/sec Superior mesenteric vein: Patent with normal hepatopetal flow. Splenic vein: Patent with reversed hepatofugal flow. Hepatic artery: Patent with a normal waveform. Hepatic artery velocity: 48 cm/sec. Hepatic artery resistive index: CBD: 2 mm. Gallbladder: Echogenic gallstone is visualized in the gallbladder neck with posterior shadowing. There is an incidental finding of multiple gallbladder polyps measuring up to 2 mm. The gallbladder wall is top normal, measuring 3 mm. There is no pericholecystic edema or sonographic Murphy sign. Right kidney length: 9.8 cm.
Right kidney appearance: Normal. Left kidney length: 11.0 cm. Left kidney appearance: Normal. Spleen: 13.2 cm, homogeneous in appearance. Large left upper quadrant varices visualized. Aorta: Normal. IVC: Normal. Other findings: None.
IMPRESSION:
cm left hepatic lobe hypoechoic vascularized mass is new since April 2016 ultrasound. There is also a new 1.8 cm right hepatic lobe hypoechoic mass. These are suspicious for hepatocellular carcinoma in the setting of HCV. Recommend additional imaging evaluation with MRI liver protocol.
2. Redemonstration of cirrhosis with mild splenomegaly and left upper cardiac and varices. 3. Shear wave measurements suggestive of F4 liver fibrosis. 4. Cholelithiasis without sonographic evidence of acute cholecystitis.
Our classifier is a weighted combination of:
Section embedding classifier: takes vector representation of the liver section recorded in US exams report as input
Lesion measure classifier: takes the two quantitative lesion measures as input
Number of lesion present in the liver
Long axis length of largest lesion
Lesions - 2
Long axis – 18 mm
Section embedding: Vectors of liver section

11. Results: LIRADS formatted reports
LI-RADS category reported by the original interpreting radiologist as the true label.
Performance of the human raters and the proposed model on the validation set (147 reports)
Two experience readers verses original image reader

12. Machine performance on LI-RADS reports
Lesion measure classifier
Section classifier
Ensemble classifier

13. Results: Reports of UT Southwestern
Trained only on Stanford data, applied on a different institutional data
Tested on 2381 reports coded with LIRADS template
Machine inferred annotation
Proposed model
Average precision
0.89
Average recall
0.84
Average f1 score
0.85
LIRADS 2 are predicted as LIRADS 3

14. Results: NON-LIRADS formatted reports
Reports formatted without LIRADS template ( ) [11,154 exams]
No LIRADS scoring available from the US image reader
Asked raters to annotated 216 reports where the model’s predicted highest probability is either <0.5 (152 reports) or >0.9 (64 reports) .
Low confidence
High confidence
Confusion Matrix on 216 reports
Prediction confidence

15. Results: Longitudinal patient tracking

16. Conclusion
We achieved 0.77 precision and 0.63 recall for non-LIRADS coded reports
Without retraining classified UT southwestern reports with 0.89 precision, recall, with an average F1 score of 0.85.
Our approach may facilitate tracking results of HCC screening examinations in patients not only across multiple years, but also between different institutions.
This paradigm supports efficient screening, recommendations and treatment planning in patients at risk for HCC.
Also, extract semantic annotation of US images by parsing free-text radiology reports

17. Email me at: imonb@stanford.edu
Thank you!
This study was supported by a grant from the Stanford/GE Blue Sky program
me at: