1. NSF MedIX REU Program Medical Imaging Projects @ DePaul CDM Daniela S. Raicu, PhD Associate Professor Email: draicu@cs.depaul.edu Lab URL: http://facweb.cs.depaul.edu/research/vc/http://facweb.cs.depaul.edu/research/vc/

2. NSF MedIX REU Program, CDM, DePaul University Outline Medical Imaging (Computed Tomography) –Content-based and semantic-based image retrieval Projects 1 and 2 –Mappings from low-level image features to semantic concepts Projects 3 and 4 –Liver segmentation Project 5

3. NSF MedIX REU Program, CDM, DePaul University - Definition of Content-based Image Retrieval: Content-based image retrieval is a technique for retrieving images on the basis of automatically derived image features such as texture and shape. Content-based medical image retrieval (CBMS) systems Applications of Content-based Image Retrieval: Teaching Research Diagnosis PACS and Electronic Patient Records

4. NSF MedIX REU Program, CDM, DePaul University Feature Extraction Similarity Retrieval Image Features [D 1, D 2,…D n ] Image Database Query Image Query Results Feedback Algorithm User Evaluation Diagram of a CBIR

5. NSF MedIX REU Program, CDM, DePaul University An image retrieval system can help when the diagnosis depends strongly on direct visual properties of images in the context of evidence-based medicine or case-based reasoning. CBIR as a Diagnosis Aid

6. NSF MedIX REU Program, CDM, DePaul University An image retrieval system will allow students/teachers to browse available data themselves in an easy and straightforward fashion by clicking on “show me similar images”. Advantages: - stimulate self-learning and a comparison of similar cases - find optimal cases for teaching Teaching files: Casimage: http://www.casimage.com http://www.casimage.com myPACS: http://www.mypacs.nethttp://www.mypacs.net CBIR as a Teaching Tool

7. NSF MedIX REU Program, CDM, DePaul University CBIR as a Research Tool Image retrieval systems can be used: to complement text-based retrieval methods for visual knowledge management whereby the images and associated textual data can be analyzed together multimedia data mining can be applied to learn the unknown links between visual features and diagnosis or other patient information for quality control to find images that might have been misclassified

8. NSF MedIX REU Program, CDM, DePaul University CBIR as a tool for lookup and reference in CT chest/abdomen Case Study: lung nodules retrieval –Lung Imaging Database Resource for Imaging Research http://imaging.cancer.gov/programsandresources/Inf ormationSystems/LIDC/page7 http://imaging.cancer.gov/programsandresources/Inf ormationSystems/LIDC/page7 –29 cases, 5,756 DICOM images/slices, 1,143 nodule images –4 radiologists annotated the images using 9 nodule characteristics: calcification, internal structure, lobulation, malignancy, margin, sphericity, spiculation, subtlety, and texture Goals: –Retrieve nodules based on image features: Texture, Shape, and Size –Find the correlations between the image features and the radiologists’ annotations

9. NSF MedIX REU Program, CDM, DePaul University LIDC Semantic Concepts Calcification1.Popcorn 2.Laminated 3.Solid 4.Non-central 5.Central 6.Absent Sphericity1.Linear 2.. 3.Ovoid 4.. 5.Round Internal structure1.Soft Tissue 2.Fluid 3.Fat 4.Air Spiculation1.Marked 2.. 3.. 4.. 5.None Lobulation1.Marked 2.. 3.. 4.. 5.None Subtlety1.Extremely Subtle 2.Moderately Subtle 3.Fairly Subtle 4.Moderately Obvious 5.Obvious Malignancy1.Highly Unlikely 2.Moderately Unlikely 3.Indeterminate 4.Moderately Suspicious 5.Highly Suspicious Texture1.Non-Solid 2.. 3.Part Solid/(Mixed) 4.. 5.Solid Margin1.Poorly Defined 2.. 3.. 4.. 5.Sharp

10. NSF MedIX REU Program, CDM, DePaul University Extracted Image Features Shape FeaturesSize FeaturesIntensity FeaturesTexture Features CircularityAreaMinIntensity 11 Haralick features calculated from co- occurrence matrices (Contrast, Correlation, Entropy, Energy, Homogeneity, 3 rd Order Moment, Inverse Differential Moment, Variance, Sum Average, Cluster Tendency, Maximum Probability) RoughnessConvexAreaMaxIntensity ElongationPerimeterMeanIntensity CompactnessConvexPerimeterSDIntensity EccentricityEquivDiameterMinIntensityBG SolidityMajorAxisLengthMaxIntensityBG ExtentMinorAxisLengthMeanIntensityBG 24 Gabor features - mean and standard deviation of Gabor filters consistency of four orientations and three scales. RadialDistanceSDSDIntensityBG IntensityDifference

11. NSF MedIX REU Program, CDM, DePaul University Lung nodule representation

12. NSF MedIX REU Program, CDM, DePaul University Choose a nodule

13. NSF MedIX REU Program, CDM, DePaul University Choose an image feature& a similarity measure

14. NSF MedIX REU Program, CDM, DePaul University Retrieved Images

15. NSF MedIX REU Program, CDM, DePaul University CBIR systems: challenges & REU projects Type of features image features: - texture features: statistical, structural, model and filter-based - shape features textual features (such as physician annotations) Project 1: Feature reduction for medical image processing - Investigate how many features with respect to the number of unique nodules - Investigate what the most important low-level image features are with respect to the retrieval process - Investigate the uniformity of the features with respect to the same type of nodules.

16. NSF MedIX REU Program, CDM, DePaul University CBIR systems: challenges & REU projects (cont.) Similarity measures -point-based and distribution based metrics Retrieval performance: precision and recall clinical evaluation Project 2: Evaluation of CBIR and SBIR systems Perform a literature review on the current techniques used to evaluate CBIR systems both for the general and medical domain Investigate ways to include radiologists’ feedback in the retrieval process Investigate ways to evaluate the retrieval process by varying various numbers of parameters such as number of images retrieved, cutoff value for acceptable precision and recall, and minimum number of radiologists/observers needed to evaluate the system.

17. NSF MedIX REU Program, CDM, DePaul University Correlations between Image Features and Concepts

18. NSF MedIX REU Program, CDM, DePaul University Automatic Mappings Extraction Step-wise multiple regression analysis was applied to generate prediction models for each characteristic c i based on all image features f k : where p is the # of used image features, are the regression coefficients, and are the prediction errors per model. Goodness of fit for the regression model:

19. NSF MedIX REU Program, CDM, DePaul University Regression Models CharacteristicsEntire dataset (1106 images, 73 nodules) At least 2 radiologists agreed At least 3 radiologists agreed Calcification0.3970.578 (884, 41)0.645 (644, 21) Internal Structure0.417- (855, 40)- (659, 22) Lobulation0.2820.559 (448, 24)0.877 (137, 6) Malignancy0.3100.641 (489, 23)0.990 (107, 5) Margin0.4030.376 (519, 28)- (245, 7) Sphericity0.2390.481 (575, 27)0.682 (207, 9) Spiculation0.3200.563 (621, 29)0.840 (228, 9) Subtlety0.3010.282 (659, 25)0.491 (360, 10) Texture0.1810.473 (736, 33)0.843 (437, 15)

20. NSF MedIX REU Program, CDM, DePaul University Texture Regression Model

21. NSF MedIX REU Program, CDM, DePaul University Malignancy Regression Model

22. NSF MedIX REU Program, CDM, DePaul University Lobulation Regression Model

23. NSF MedIX REU Program, CDM, DePaul University Spiculation Regression Model

24. NSF MedIX REU Program, CDM, DePaul University Image Features – Semantics Mappings: challenges & REU projects Project 3: Multi-view learning classifier for lung nodule classification Investigate which image features are the best for individual semantic characteristics, build classifiers for each one of the individual classifiers, and combine the individual classifies for optimal learning/classification of lung nodules Project 4: Bridging the semantic gap in lung nodule interpretation Investigate ways to clinically evaluate the mappings from low-level image features to semantic characteristics Investigate the effect of the imaging acquisition parameters (such as pitch, FOV, and reconstruction kernel) on the proposed mappings

25. NSF MedIX REU Program, CDM, DePaul University - Pixel-level Classification: - tissue segmentation - context-sensitive tools for radiology reporting Liver Segmentation in CT images Pixel Level Texture Extraction Pixel Level Classification Organ Segmentation

26. NSF MedIX REU Program, CDM, DePaul University Liver Segmentation in CT images Example of Liver Segmentation: (J.D. Furst, R. Susomboon, and D.S. Raicu, "Single Organ Segmentation Filters for Multiple Organ Segmentation", IEEE 2006 International Conference of the Engineering in Medicine and Biology Society (EMBS'06)) Region growing at 70%Region growing at 60%Segmentation Result Original ImageInitial Seed at 90%Split & Merge at 85%Split & Merge at 80%

27. NSF MedIX REU Program, CDM, DePaul University Liver Segmentation using Automatic Snake a) b)c) d) Figure: a) Gradient vector flow segmentation; b) Traditional vector field segmentation; c) and,d) Respective segmentations overlaid on ground truth (white). a) Project 5: Automatic selection of initial points for snake- based segmentation

28. NSF MedIX REU Program, CDM, DePaul University uestions ?