1. Medical Imaging Projects Daniela S. Raicu, PhD Assistant Professor Email: draicu@cs.depaul.edu Lab URL: http://facweb.cs.depaul.edu/research/vc/http://facweb.cs.depaul.edu/research/vc/

2. MedIX REU Program, Summer 2007 2 IMP & MediX Labs @ DePaul Faculty: GM. Besana, L. Dettori, J. Furst, G. Gordon, S. Jost, D. Raicu, N. Tomuro CTI Students: W. Horsthemke, C. Philips, R. Susomboon, J. Zhang E. Varutbangkul, S.G. Valencia IMP Collaborators & Funding Agencies National Science Foundation (NSF) - Research Experience for Undergraduates (REU) Northwestern University - Department of Radiology, Imaging Informatics Section University of Chicago – Medical Physics Department Argonne National Laboratory - Biochip Technology Center MacArthur Foundation

3. MedIX REU Program, Summer 2007 3 Outline  Medical Imaging and Computed Tomography  Soft Tissue Segmentation in Computed Tomography  Project 1: Region-based classification  Project 2: Texture-based snake approach  Content-based Image Retrieval and Annotation  Project 3: Lung Nodule Retrieval based on image content and radiologists’ feedback  Project 4: Associations discovery between image content and radiologists’ assessment

4. MedIX REU Program, Summer 2007 4 The study of medical imaging is concerned with the interaction of all forms of radiation with tissue and the development of appropriate technology to extract clinically useful information from observation of this technology. What is Medical Imaging (MI)? X-RayfMRI CT

5. MedIX REU Program, Summer 2007 5 _______________________________________________ Computed Tomography (CT) G. Hounsfield (computer expert) and A.M. Cormack (physicist) (Nobel Prize in Medicine in 1979) CT overcomes limitations of plain radiography CT doesn’t superimpose structures (like X-ray) CT is an imaging based on a mathematical formalism that states that if an object is viewed from a number of different angles than a cross-sectional image of it can be computed (reconstructed)

6. MedIX REU Program, Summer 2007 6 Stages of construction of a voxel dataset from CT data (a)CT data capture works by taking many one dimensional projections through a slice (scanning) (b) CT reconstruction pipeline CT Data

7. MedIX REU Program, Summer 2007 7 _______________________________________________ CT – Data Acquisition Slice-by-slice acquisition X-ray tube is rotating around patient to acquire a slice patient is moved to acquire the next slice Volume acquisition X-ray tube is moving continuously along a spiral (helical) path and the data is acquired continuously

8. MedIX REU Program, Summer 2007 8 (a)slice-by-slice scanning (b) Spiral (volume) scanning CT – Data Acquisition

9. MedIX REU Program, Summer 2007 9 CT – SPIRAL SCANNING a patient is moved 10mm/s (24cm / single scan) slice thickness: 1mm-1cm faster than slice-by-slice CT no shifting of anatomical structures slice can be reconstructed with an arbitrary orientation with (a single breath) volume CT multi-slice systems: parallel system of detectors 4/8/16 slices at a time generates a large data of thin slices better spatial resolution (  better reconstruction)

10. MedIX REU Program, Summer 2007 10 Understanding Visual Information: Technical, Cognitive and Social Factors CT - DATA PROCESSING CT numbers (Hounsfield units) HU: computed via reconstruction algorithm (~tissue density/ X-ray absorption) most attenuation (bone) least attenuation (air) blood/calcium increases tissue density

11. MedIX REU Program, Summer 2007 11 Understanding Visual Information: Technical, Cognitive and Social Factors Relationship between CT numbers and brightness level CT - DATA PROCESSING

12. MedIX REU Program, Summer 2007 12 CT - IMAGE DISPLAY Thoracic image: a) width 400HU/level 40HU (no lung detail is seen) b) width 1000HU/level –700HU (lung detail is well seen; bone and soft tissue detail is lost) Human eye can perceive only a limited range gray-scale values

13. MedIX REU Program, Summer 2007 13 CT Medical Imaging (MI)@ CTI Filtering Correction Registration Segmentation Analysis VisualizationClassificationRetrieval Projects 1&2: Texture-based soft-tissue segmentation Projects 3&4: Content-based medical image retrieval and annotation

14. MedIX REU Program, Summer 2007 14 Outline  Medical Imaging and Computed Tomography  Soft Tissue Segmentation in Computed Tomography  Project 1: Region-based classification approach  Project 2: Texture-based snake approach  Content-based Image Retrieval and Annotation  Project 3: Lung Nodule Retrieval based on image content and radiologists’ feedback  Project 4: Associations discovery between image content and radiologists’ assessment

15. MedIX REU Program, Summer 2007 15 Goal: context-sensitive tools for radiology reporting Approach: pixel-based texture classification Soft-tissue Segmentation in Computed Tomography Pixel Level Texture Extraction Pixel Level Classification Organ Segmentation

16. MedIX REU Program, Summer 2007 16 Pixel-based texture extraction: Soft-tissue Segmentation in Computed Tomography Pixel Level Texture Extraction Challenges:  Storage:  Input: 0.5+ terabyte of raw data dispersed over about 100K+ images  Output: 90+ terabytes of low-level features in a 180 dimensional feature space  Compute:  24 hours of compute time = 180 features for a single image on a modern 3GHz workstation Input Patient Data Characteristics:  hundreds of images per patient  image spatial resolution: 512 x512  image gray-level resolution: 2 12 Output Data Characteristics:  low-level image features (numerical descriptors)  k=180 Haralick texture features per pixel (9 descriptors x4 directions x5 displacements)

17. MedIX REU Program, Summer 2007 17 Project 1: Challenges and opportunities  Calculate image features at region-level instead of pixel-level  Include Gabor features in the feature extraction phase in addition to the co- occurrence texture features  Explore different approaches for region classification in addition to the decision tree approach Current Implementation: Matlab Stack of CT slicesImage Partitioning Feature Extraction Region Classification

18. MedIX REU Program, Summer 2007 18 Liver Segmentation Example J.D. Furst, R. Susomboon, and D.S. Raicu, "Single Organ Segmentation Filters for Multiple Organ Segmentation", IEEE 2006 International Conference of the Engineering in Medicine and Biology Society (EMBS'06) Region growing at 70%Region growing at 60%Segmentation Result Original ImageInitial Seed at 90%Split & Merge at 85%Split & Merge at 80%

19. MedIX REU Program, Summer 2007 19 Snake Application Demo Next figures are demonstrated how to automatically classify the CT images of heart and liver. Soft-tissue Segmentation in Computed Tomography

20. MedIX REU Program, Summer 2007 20 Demo For HEART There are 4 main menu to operate this application. OPEN: To open a new Image. SEGMENT: To automatically segment the region of interest organ TEXTURE: To calculate the texture models: co- occurrence/ run-length CLASSIFICATION: To automatically classify the segmented organ

21. MedIX REU Program, Summer 2007 21 HEART: Segmentation The application allows users to change Snake/ Active contour algorithm parameters

22. MedIX REU Program, Summer 2007 22 HEART: Segmentation (cont.) Button is clicked User selects points around the region of interest

23. MedIX REU Program, Summer 2007 23 HEART: Segmentation (result) Show segmented organ If the user likes the result of the segmentation, then the user will go to the classification step

24. MedIX REU Program, Summer 2007 24 HEART: Classification Selection of texture models: Co-occurrence, Run-length, Or Combine both models Texture features corresponding to the selected texture model are calculated and shown here

25. MedIX REU Program, Summer 2007 25 HEART: Classification Result Results are shown as follows. Predicted organ: Heart Probability:0.86 And also rule which is used to predict that this segmented organ is HEART

26. MedIX REU Program, Summer 2007 26 Demo For LIVER Start application by open and load the image.

27. MedIX REU Program, Summer 2007 27 LIVER: Segmentation The application allows users to change Snake/ Active contour algorithm parameters

28. MedIX REU Program, Summer 2007 28 LIVER: Segmentation (cont.) Segmentation Button is clicked User selects points around the region of interest

29. MedIX REU Program, Summer 2007 29 LIVER: Segmentation Result Show segmented organ If user is satisfied with the result, then it will go to the classification step

30. MedIX REU Program, Summer 2007 30 LIVER: Classification Select texture models: Co-occurrence, Run-length, Or Combine both models Texture features is calculated for the selected model

31. MedIX REU Program, Summer 2007 31 LIVER: Classification Result Results are shown as follows. Predicted organ: Liver Probability:1.00 And also rule which is used to predict that this segmented organ is LIVER

32. MedIX REU Program, Summer 2007 32 Project 2: Challenges and opportunities  Calculate texture image features at the pixel level instead of using the gray- levels  Apply snake on the texture features  Investigate different ways to objectively compare two segmentation algorithms, in particular the snake and the classification-based approach Current Implementation: Matlab

33. MedIX REU Program, Summer 2007 33 Outline  Medical Imaging and Computed Tomography  Soft Tissue Segmentation in Computed Tomography  Project 1: Region-based classification approach  Project 2: Texture-based snake approach  Content-based Image Retrieval and Annotation  Project 3: Lung Nodule Retrieval based on image content and radiologists’ feedback  Project 4: Associations discovery between image content and radiologists’ assessment

34. MedIX REU Program, Summer 2007 34 Outline  Medical Imaging and Computed Tomography  Soft Tissue Segmentation in Computed Tomography  Project 1: Region-based classification approach  Project 2: Texture-based snake approach  Content-based Image Retrieval and Annotation  Project 3: Lung Nodule Retrieval based on image content and radiologists’ feedback  Project 4: Associations discovery between image content and radiologists’ assessment

35. MedIX REU Program, Summer 2007 35 - Definition of Content-based Image Retrieval: Content-based image retrieval is a technique for retrieving images on the basis of automatically derived image features such as texture and shape. Content-based medical image retrieval (CBMS) systems Applications of Content-based Image Retrieval:  Teaching  Case-base reasoning  Evidence-based medicine

36. MedIX REU Program, Summer 2007 36 Feature Extraction Similarity Retrieval Image Features [D 1, D 2,…D n ] Image Database Query Image Query Results Feedback Algorithm User Evaluation Diagram of a CBIR

37. MedIX REU Program, Summer 2007 37 CBIR as a tool for lookup and reference Case Study: lung nodules retrieval –Lung Imaging Database Resource for Imaging Research http://imaging.cancer.gov/programsandresources/Inf ormationSystems/LIDC/page7 http://imaging.cancer.gov/programsandresources/Inf ormationSystems/LIDC/page7 –29 cases, 5,756 DICOM images/slices, 1,143 nodule images –4 radiologists annotated the images using 9 nodule characteristics: calcification, internal structure, lobulation, malignancy, margin, sphericity, spiculation, subtlety, and texture Goals: –Retrieve nodules based on image features: Texture, Shape, and Size –Find the correlations between the image features and the radiologists’ annotations

38. MedIX REU Program, Summer 2007 38 Examples of nodule images

39. MedIX REU Program, Summer 2007 39 CBIR as a tool for lung nodule lookup and reference Low-level feature extraction:

40. MedIX REU Program, Summer 2007 40 Nodule Characteristics –Calcification (1. Popcorn, 2. Laminated, 3. Solid, 4. Non-Central, 5. Central, 6. Absent) –Internal Structure (1. soft tissue, 2. fluid, 3. fat, 4. air) –Subtlety (1. extremely subtle,..................., 5. obvious) –Sphericity (1. Linear, 2......., 3. Ovoid, 4......, 5. Round) –Texture (1. Non-Solid, 2......, 3. Part Solid, 4........, 5. Solid)

41. MedIX REU Program, Summer 2007 41 Nodule Characteristics –Margin (1. Poorly,......................., 5. Sharp) –Lobulation (1. Marked,....................., 5. No Lobulation) –Spiculation (1. Marked,....................., 5. No Spiculation) –Malignancy (1. Highly Unlikely for Cancer,..............., 5. Highly Suspicious for Cancer)

42. MedIX REU Program, Summer 2007 42 Choose a nodule

43. MedIX REU Program, Summer 2007 43 Choose an image feature& a similarity measure M. Lam, T. Disney, M. Pham, D. Raicu, J. Furst, “Content-Based Image Retrieval for Pulmonary Computed Tomography Nodule Images”, SPIE Medical Imaging Conference, San Diego, CA, February 2007

44. MedIX REU Program, Summer 2007 44 Retrieved Images

45. MedIX REU Program, Summer 2007 45 Project 3: Challenges and opportunities  Calculate co-occurrence texture features at the local level instead of global level  Incorporate shape and size features in the retrieval process in addition to texture features  Integrate radiologists’ assessments/feedback into the retrieval process  Investigate different approaches for retrieval in addition to similarity measures  Report the retrieval results with a certain confidence level (probability) instead of just a binary output (similar/not similar) Current implementation: C# Available Open Source at: http://brisc.sourceforge.net/http://brisc.sourceforge.net/

46. MedIX REU Program, Summer 2007 46 Outline  Medical Imaging and Computed Tomography  Soft Tissue Segmentation in Computed Tomography  Project 1: Region-based classification approach  Project 2: Texture-based snake approach  Content-based Image Retrieval and Annotation  Project 3: Lung Nodule Retrieval based on image content and radiologists’ feedback  Project 4: Associations discovery between image content and radiologists’ assessment

47. MedIX REU Program, Summer 2007 47 Associations between image content and semantics

48. MedIX REU Program, Summer 2007 48 Project 4: Challenges and opportunities  Investigate other approaches for finding the associations between image features and radiologists’ assessment in addition to logistic regression and decision trees  from image content to semantics  from semantics to semantics  from image features and semantics to semantics  Create GUIs to display examples of images for each semantic concept  Investigate how the current associations discovery approaches apply to mammography assessment (Northwestern project) Current implementation: Matlab, Weka, SPSS

49. MedIX REU Program, Summer 2007 49 Questions? Thank you!