Texture Analysis for Pulmonary Nodules Interpretation and Retrieval By Ryan Datteri MEDIX program Advisors: Dr. Daniela Raicu Dr. Jacob Furst

Motivation Increase the accuracy of medical diagnostics Give radiologists comparable and similar images Make diagnostics less subjective Discover the best texture features to use for pulmonary nodule retrieval 11/17/2018

BRISC Content Based Image Retrieval System (CBIRS) Four Retrieval methods implemented Gabor Markov Local Co-occurrence Global Co-occurrence Methods are all texture based 11/17/2018

Diagram of BRISC 11/17/2018

Texture Retrieval Gray Level Images Diversity of shapes and sizes of nodules Use texture to retrieve similar nodules Precision: # of retrieved instances of the query nodule # of relevant images 11/17/2018

Similarity Measures For histogram comparison For vector comparison Jeffrey-Divergence Chi-Square For vector comparison Euclidean Manhattan Chebychev 11/17/2018

Local Co-occurrence Variables Intensity bins Allow statistical relevance to appear in the co-occurrence matrices Histogram bins Used to compute the similarity measure Window Size Subset to perform co-occurrence around each relevant pixel 11/17/2018

Local Co-occurrence Testing Window sizes 5x5, 7x7, 9x9, 11x11, and 13x13 Number of bins for intensity 64, 128, 256, and 512 Number of bins for the histogram 8, 16, 32, 64, 80, and 96. 11/17/2018

Best Found Attributes 2020 images queried 149 unique nodules Window Size = 5x5 Number of histogram bins = 96 Image Binning = 64 bins for intensity 11/17/2018

LIDC Diagram Case/ Patient Nodule 1 Nodule 2 Slice 1 Slice 2 Slice 3 Radiologist rating 1 Radiologist rating 2 Contour 2 Contour 3 Contour 1 11/17/2018

Evaluation Method #1 Precision = # of retrieved instances of the query nodule # of retrieved images 11/17/2018

Results: Local Co-occurrence 11/17/2018

Results: Local vs. Global 11/17/2018

Texture Model Comparison 11/17/2018

Image Size and Radiologist Agreement See the performance of the texture models on larger images Radiologist Agreement See which texture models are more attune to a radiologists’ perceptions 11/17/2018

Image Size 11/17/2018

Radiologist Agreement 11/17/2018

Combination of Features Used all texture models Gabor filters Markov Random Fields Local Co-occurrence Global Co-occurrence Optimal similarity measure used 11/17/2018

Combination of Features Results 11/17/2018

Principal Component Analysis Reduces features Combined pixel based texture models Gabor filters MRF Local Co-occurrence Included components that accounted for 99% variability in data Reduced data to 25 Components from 3712 features (.67% of original size) 11/17/2018

PCA Results: 1 Item Retrieved 11/17/2018

PCA Results 11/17/2018

PCA^2 Performed PCA on individual texture models Reduced Feature Set = Components that accounted for 99% variability Applied PCA to the resulting data set Reduced Data to 119 Components from 3712 features (3.2% of the original feature set size) 11/17/2018

PCA^2 Results 11/17/2018

Evaluation Method #2 Evaluate using radiologists’ annotations Results should relate to radiologists’ perceptions Sort the database based on the radiologists’ annotations using Jaccard’s similarity Sort another list using a texture model Assign values from n to 1 (n to the first item in the list, n - 1 to the second, etc . . .) to the lists 11/17/2018

#2 Continued Use small number for n Radiologists will only look at small number of retrieved results Assigned values multiplied with value of the corresponding image in the other list Best case when Lists are exactly the same: Worst Case, no matching item in either list: 0 11/17/2018

Diagram of Evaluation Method #2 Annotation Ground Truth Example. Results in a value of 379 and 98.44% Precision ‘Distance’ for left list = value computed from Jaccard’s similarity on radiologists’ annotations ‘Distance’ for right list = value from the similarity measure used for a specifice texture model Score = reversed rank 11/17/2018

Texture Models Evaluated Using Method #2 11/17/2018

Texture Models Evaluation Methods #1 #2 1. All Features 2. Gabor 2. Markov 3. Markov 3. Gabor 4. Local 5. Global Texture Models perform in nearly the same rank as Evaluation Method 1 Reaffirms Evaluation Method 1 Some correlation between Radiologists’ perceptions and our texture models 11/17/2018

Radiological Agreement 11/17/2018

Similarity Measures Evaluated Using the Second Method Global Local Gabor Markov Euclidean 25.75% 30.34% 26.40% Manhattan 25.61% Chebychev 25.43% Jeffrey Div. 30.25% 30.81% 24.40% Chi-Square 31.55% 32.68% 11/17/2018

Conclusions Local Co-occurrence performs better then Global Co-occurrence Local Co-occurrence performs comparably to Gabor and Markov Combination of Textures Performed the best PCA on all features resulted in .67% of the original features with a 3% loss in precision. PCA on each separate texture model and then on the combination of the resulting components had a 1% loss in precision with less features Evaluation methods are valid Evaluation #2 allows comparison not ‘how’ 11/17/2018

Conclusions Texture Models Evaluation Method 1 Evaluation Method 2   Texture Models Evaluation Method 1 Evaluation Method 2 Radiologist Annotation (Eval. #2) 4 Agree % Behind The Best 1 All Features 91.14 33.64 70.14 0% 2 MRF 87.00 32.68 70.42 -4.82% 3 Gabor 88.00 31.55 67.17 -8.20% 4 Local Co-occurrence 64.21 30.25 70.89 -29.57% 11/17/2018

Conclusions Texture Models Evaluation Method 1 Evaluation Method 2   Texture Models Evaluation Method 1 Evaluation Method 2 Radiologist Annotation (Eval. #2) 4 Agree % Behind The Best 1 All Features 91.14 33.64 70.14 0% 2 MRF 87.00 32.68 70.42 -4.82% 3 Gabor 88.00 31.55 67.17 -8.20% 4 Local Co-occurrence 64.21 30.25 70.89 -29.57% 11/17/2018

Conclusions Texture Models Evaluation Method 1 Evaluation Method 2   Texture Models Evaluation Method 1 Evaluation Method 2 Radiologist Annotation (Eval. #2) 4 Agree % Behind The Best 1 All Features 91.14 33.64 70.14 0% 2 MRF 87.00 32.68 70.42 -4.82% 3 Gabor 88.00 31.55 67.17 -8.20% 4 Local Co-occurrence 64.21 30.25 70.89 -29.57% 11/17/2018

Conclusions Texture Models Evaluation Method 1 Evaluation Method 2   Texture Models Evaluation Method 1 Evaluation Method 2 Radiologist Annotation (Eval. #2) 4 Agree % Behind The Best 1 All Features 91.14 33.64 70.14 0% 2 MRF 87.00 32.68 70.42 -4.82% 3 Gabor 88.00 31.55 67.17 -8.20% 4 Local Co-occurrence 64.21 30.25 70.89 -29.57% 11/17/2018

Next Steps Clustered Searching Textons Combine all features with Radiologists’ Annotations See how the results improve Incorporate relevance feedback Clustered Searching Textons Develop the ‘how’ for evaluation #2 11/17/2018

References 1. M. Lam, T. Disney, M. Pham, D. Raicu, J. Furst, R. Susomboon, “Content-Based Image Retrieval for Pulmonary Computed Tomography Nodule Images”, SPIE Medical Imaging Conference, San Diego, CA, February 2007. 2. Cancer Facts and Figures, American Cancer Society, 2006. 3. What are the Key Statistics about Lung Cancer? American Cancer Society, 2006. http://www.cancer.org/docroot/CRI/content/CRI_2_4_1x_What_Are_the_Key_Statistics_About_Lung_Cancer_15.asp?sitearea. 4. A. Corboy, W. Tsang, D. Raicu, J. Furst, "Texture-Based Image Retrieval for Computerized Tomography Databases", IEEE - Transactions on Information Technology in Biomedicine, 2006. 5. A. Materka and M.Strzelecki, “Texture analysis methods – a review,” tech. rep., Technical University of Lodz, Institute of Electronics, 1998. COST B11 report. 6. T. Andrysiak and M. Choras, “image retrieval based on hierarchical gabor filters,” International Journal Applied Computer Science 15 (4), pp. 471-480, 2005. 7. C. Chen, L. Pau, and P. W. (eds.), The Handbook of Pattern Recognition and Computer Vision (2nd Edition), World Scientific Publishing Company, 1998. 8. Malik, J., Belongie, S., Shie, J., and Leung, T. 1999. “Textons, contours and regions: Cue integration in image segmentation. In Proc. IEEE Intl. Conf. Computer Vision, Vol. 2, Corfu, Greece, pp. 918-925. 9. C. Wei, C. LI, R. Wilson. “A General Framework for Content-Based Medical Image Retrieval with its application to Mammograms”, Proc. SPIE Medical Imaging 2005, April 2005; 134-143. 10. S. K. Kinoshita, P.M.  de Azevedo-Marques, R.R. Pereira Júnior, J.A.H. Rodrigues, and R.M. Rangayyan,  “Content-based retrieval of mammograms using visual features related to breast density patterns”. To appear in Journal of Digital Imaging, 2007. 11. C.-R. Shyu, C. Brodley, A. Kak, A. Kosaka, A. M. Aisen, and L. S. Broderick, “Assert: A physician-in-the-loop conten-based retrieval system for hrct image databases,” Computer Vision and Image Understanding 75, pp. 111-132, July/August 1999. 11/17/2018

Special Thanks DePaul University MedIX Program National Science Foundation Dr. Raicu Dr. Furst 11/17/2018

Questions???? 11/17/2018