An Investigation into the Relationship between Semantic and Content Based Similarity Using LIDC Grace Dasovich Robert Kim Midterm Presentation August

Outline Related Work Data Modeling Approach and Results –Similarity Measures –Artificial Neural Network –Multivariate Linear Regression Conclusions Future Work

Computer-Aided Diagnosis (CADx) based on low-level image features –Armato et al. developed a linear discriminant classifier using features of lung nodules –Need to find the relationship between the image features and radiologists’ ratings Related Work

Image features and the semantic ratings –Lung Interpretations Barb et al. developed Evolutionary System for Semantic Exchange of Information in Collaborative Environments (ESSENCE) Raicu et al. used ensemble classifiers and decision trees to predict semantic ratings Samala et al. used several combinations of image features and the radiologists’ ratings to classify nodules Related Work

–Similarity Li et al. investigated four different methods to compute similarity measures for lung nodules –Feature-based –Pixel-value-difference –Cross correlation –ANN Related Work

Materials LIDC Dataset 149 Unique Nodules –One slice per nodule, largest nodule area 9 Semantic Characteristics –Calcification and Internal Structure had little variation, thus were not used 64 Content Features –Shape, size, intensity, and texture 6 Data

Related Work Data Modeling Approach and Results –Similarity Measures –Artificial Neural Network –Multivariate Linear Regression Conclusions Future Work Outline

Cosine Similarity Jeffrey Divergence Euclidean Distance Similarity Measures

Computed feature distance measures Similarity Measures

Outline Related Work Data Modeling Approach and Results –Similarity Measures –Artificial Neural Network –Multivariate Linear Regression Conclusions Future Work

Two three-layer ANNs –Input (64 neurons), hidden layer (5 neurons), output (1) –Input (64 neurons), hidden layer (5 neurons), output (7) Input = 64 feature distances Output = Semantic similarity or difference in semantic ratings Hyperbolic tangent function, backpropagation algorithm, 200 iterations Methods

ANN with a single output –640 random pairs from all 109 nodules –231 pairs from nodules with malignancy > 3 –496 pairs from nodules with area > 122 mm 2 Methods

ANN with seven outputs –640 random pairs from all 109 nodules

Leave-one-out method –Cosine similarity or Jeffrey divergence or difference in Semantic ratings used as teaching data –An ANN trained with entire dataset minus one image pair –The pair left out used for testing –Correlation between calculated radiologists’ similarity and ANN output calculated Methods

ANN with a single output –640 random pairs from all 109 nodules –231 pairs from nodules with malignancy > 3 –496 pairs from nodules with area > 122 mm 2 ANN with seven outputs –640 random pairs from all 109 nodules Methods

ANN using 640 random pairs Results

ANN using 231 pairs with malignancy rating > 3 Results

ANN using 496 pairs with area > 122 mm 2 Results

ANN output vs. target values using Jeffrey divergence for the 640 pairs (r = 0.438) Results

ANN using random 640 pairs and the Jeffrey divergence with seven semantic ratings Results

Outline Related Work Data Modeling Approach and Results –Similarity Measures –Artificial Neural Network –Multivariate Linear Regression Conclusions Future Work

Methods Normalization of Features –Min-Max Technique –Z-Score Technique Pair Selection –Looked for matches between k number of most similar images based on semantic and content 24 Methods

Multivariate Regression Analysis –Select features with highest correlation coefficients –Feature distance measures 25 Methods

Nodule Analysis –Determine differences between selected and non-selected nodules –Define requirements for our model Methods

Results 27 Results

d(i, j)d 2 (i, j)exp(d(i, j)) Cosine Jeffrey

Results Correlation CoefficientFeature Equivalent Diameter Energy (Haralick) Gabor Mean 135_ Convex Area Gabor STD 135_ Min Intensity BG Markov Variance (Haralick) Gabor STD 45_ SD Intensity R 2 = Results

30 Results

31 Results

32 Results

A. Equivalent Diameter, B. Standard Deviation of Intensity, C. Malignancy, D. Subtlety

Preliminary Issues The ANN also is not yet sufficient to predict semantic similarity from content –Best correlation –Malignancy correlation –Jeffrey performed better unlike linear model A semantic gap still exists Conclusions

Our linear model applies to a specific type of nodule –Characteristics: High malignancy, high texture, low lobulation, and low spiculation –Features: Larger diameter, greater intensity Linear models are not sufficient for determination of similarities –R 2 of with chosen nodules 35 Conclusions

Future Work Reduce variability among radiologists –Use only nodules with radiologists’ agreement Find best combination of content features –64 may be too many –Currently only using 2D Future Work

Different semantic distance measures –Some ratings are ordinal, Jeffery is for categorical Different methods of machine learning –Incorporate radiologists’ feedback into training –Ensemble of classifiers Future Work

Thanks for Listening Any Questions? 38 Thanks for Listening