ASSERT: A Physician-in-the-loop Content-based Retrieval Images for HRCT Image Databases Authors: C. Shyu, C.Brodley, A. Kak, A. Kosaka, A. Aisen, L. Broderick Journal: Computer Vision and Image Understanding , Vol. 75, 1999

ASSERT: A Physician-in-the-loop Content-based Retrieval (CBIR) Images for HRCT Image Databases ASSERT: A CBIR system for High-Resolution Computed Tomography of the lung in which the physician is involved in both the delineation of pathology bearing regions and the evaluation of the retrieval accuracy. ASSERT can be extended to the CT of the liver and the MRI of the knee A collaborative project: The Department of Radiology at Indiana University The School of Medicine at University of Wisconsin School of Electrical and Computer Engineering at Purdue University 4/6/2019

Goal of the ASSERT System To retrieve the most similar images with a query image in a HRCT lung database: To pose a query, the physician circles one or more pathology bearing regions (PBR) in the query image. The system then retrieves the most n similar images from the database using an index comprised of localized features of the PBRs and of the global image. Improve retrieval precision by using user relevance feedback 4/6/2019

Importance of the System CBIR is needed for the lung diagnosis because the current art in diagnosis, for those cases not immediately recognizable, is to consult a published atlas of lung pathologies. ASSERT saves the radiologist from the laborious task of paging through the atlas looking for an image that matches the pathology of the current patient. 4/6/2019

What is a CBIR system? Image retrieval systems that permit image searching based on features automatically extracted from the images’ own visual content are called content-based image retrieval (CBIR) systems. Domain-specific features: - fingerprints, human faces, lobular features Visual features (primitive or low-level image features –local or global features) General features: - color, texture, shape High level features: keywords/concepts 4/6/2019

Diagram of a CBIR System Useful content in medical images consists of gray level variations on highly localized regions in the images and it is very hard to automatically extract these regions by image segmentation techniques The retrieval results have to correspond to the human/physician perception of the image similarity ? 4/6/2019

Example of HRCT lung Image & PBR delineated by physicians Feature extraction: I. The Physician delineates the Pathology Bearing Regions (PBRs) and a set of anatomical landmarks (fissures); II. Information regarding the PBR of the lung resides as much on the location of the PBR as it does in the visual content of the PBR. 4/6/2019

Location of the PBRs: Feature extraction: III. Location of the PBRs: Interior of a lobular region Adjacent to a lobular region Lobular region is defined by the lung boundary and the fissures that are present in the lung. These fissures are delineated by the physician when visible. 4/6/2019

Lobular Regions: 4/6/2019

Lobular Feature Set (LFS) A LFS consists of a lobe or a combination of adjacent lobes and the PBRs found therein. ASSERT Assumption: there are at most two PBRs within a lobe and one is adjacent to the boundary and the other one is interior to the lobe. If each PBR is represented by a N dimensional vector, a 2N dimensional space is constructed to represent a LFS: ASSERT: N = 26, 14 attributes (local to PBRs) are related to the characterization of the perceptual categories used in lung diseases, and the other 12 attributes for general purpose characterization of the entire lung region. 4/6/2019

Image/PBS Characterization Bronchiectasis Peripheral honeycombing Subpleural interstitial Thickening, etc 4/6/2019

Example of Disease, Perceptual Category and Low-level Attributes Ex: Bronchiectasis: thickening the walls of the bronchi (air-filled passages in the lung that due to their low x-ray attenuation show up as dark regions. - Thickness of the bronchi-walls and - the sizes of pulmonary arteries adjacent to bronchi Linear and reticular Opacities (thin and filament- like elements) 4/6/2019

The extraction of low-level features (local to PBR) corresponding to perceptual categories For linear and reticular opacities: a “dual thresholding” scheme (Otsu’s algorithm) in which a high threshold is first used to extract the dark regions and then another high threshold is applied to the inverted image to extract the white regions surrounding the dark regions. The resulting pairs of dark and white regions corresponding to the walls and the lumen of the bronchi are accepted only if their centers of mass are co-located to within a tolerance value. For each pair, the following attributes are extracted: Thickness of the bronchi-walls Sizes of pulmonary arteries adjacent to bronchi 4/6/2019

The extraction of general purpose low-level features General purpose attributes local to PBRs to characterize the shape, texture and other gray-level attributes General purpose attributes global to the entire lung region There are 255 general purpose attributes which, through a sequential forward search (SFS) algorithm, are reduced to only 12 attributes. Therefore, a medical image is represented by a Lobular Feature Set (LFS) whose structure is a 2N dimensional feature vector (N=26=12+14) 4/6/2019

Diagram of a CBIR System LFS vectors LFS classes: Multi-Hashing Two LFSs in two separate images belong to The same LFS class if the associated PBRs are in the same adjacency/interior relationship with respect to the lobular boundaries and if the PBRs carry the same diagnostic information. 4/6/2019

The Retrieval Stage Candidate classes: on average, 2.63 LFS classes After applying filter 1: the remaining LFS classes point to 25 images After applying filter 2: the four best images are selected for display. 4/6/2019

Experimental Results Medical Image Database: 302 images from 78 patients Images were labeled by physicians; 8 diseases A retrieval is considered successful if all four retrieved images have the same label Retrieval accuracy is measured by the precision coefficient Retrieval efficiency is measured by the speed of retrieval Sensitivity to the Physician Subjectivity 4/6/2019

Experimental Results: Local versus Global Attributes R1(p)=attributes extracted from the PBR region C=attributes contrasting the PBR with respect to the rest of the lung region R2(G)=attributes for the entire lung region 4/6/2019

Sensitivity to Physician Subjectivity 4/6/2019

Retrieval Accuracy and Efficiency Pno_loc= precision achieved with the retrieval of four best matches whose PBRs have the same disease labels as the query image, regardless of the adjacency/interior attributes of the PBRs. Speedup=(Total number of database images)/(Number of candidate Images) 4/6/2019