Addressing the Medical Image Annotation Task using visual words representation Uri Avni, Tel Aviv University, Israel Hayit GreenspanTel Aviv University, Israel Jacob GoldbergerBar Ilan University, Israel

Outline o Challenge description o Proposed system o Image representation o classification o Results o Parameters optimization o Performance analysis o Conclusion

ImageClef 2009 medical annotation challenge 12,677 classified x-ray images, 1733 unknown images Classification according to four labeling sets: o 57 classes o 116 classes o 116 IRMA codes o 196 IRMA codes

Noisy images Irregular brightness, contrast Non-uniform class distribution IRMA database The IRMA group - Aachen University of Technology (RWTH), Germany

Great intra-class variability Category #: Sagittal, Mediolateral, Left hip IRMA Database - samples

Category # overview image posteroanterior (PA) Category # high beam energy posteroanterior (PA),expiration Category # high beam energy anteroposterior (AP),inspiration Category # overview image anteroposterior (AP), supine IRMA Database - samples Great inter-class similarity

Outline o Challenge description o Proposed system o Image representation o classification o Results o Parameters optimization o Performance analysis o Conclusion

Image representation o Move from 2D image to a vector of numbers o Representation should preserve enough information of the image content o Should be not sensitive to translation, artifacts and noise o Compare and classify the compact representation Word number Image model

Patch extraction Extract raw pixels from patches of fixed size Dense sampling, ~200,000 patches per image Normalize intensity, variance Ignore empty patches Sample several images – one collection with millions of patches

Feature space description - Reduce dimension of the collection - Add position (x,y) to the features, position weight is important - 8 dimensional feature vector 9x9 pixels PCA 6 coefficients

Build dictionary Select k feature vectors as far apart as possible Run k-means clustering Cluster centers, with x,y Cluster centers

Image representation Scan image – translate patches to words histogram Image Dictionary Word number Probability

Image representation Use multiple scales

Classification Examine knn classifier, with different distance metrics Examine several SVM kernels: Radial basis function Chi-square Histogram intersection One-vs-one multiclass SVM classifier, with n(n-1)/2 binary classifiers

Outline o Our objective o Proposed system o Image representation o Retrieval & classification o Results o Parameters optimization o Performance analysis o Conclusion and future work

Selecting classifier type Effect of histogram distance metric in k-nearest neighbors vs svm classifier SVM Symmetric Kullback – Leibler divergence Jeffery divergence

Selecting feature space Effect of parameters on classification accuracy, using 20 cross-validation experiments with x,y No x,y

Selecting type of features - invariance / discriminative power tradeoff Selecting features Feature typeAverage % correctStandard dev Raw patches SIFT* Normalized Patches * Scale and rotation invariance are not desired

Running time 12,677 images Running on Intel daul quad core Xeon 2.33Ghz Build dictionary Extract features Train classifier classification time per image Total (train + classify) Raw6 min96.8 min6 min0.54 sec126 min SIFT10 min597 min6 min3.32 sec724 min

Selecting dictionary

Using multiple dictionaries for 3 scales increases classification accuracy by 0.5%

Classification results – effect of kernel Effect of kernel function on SVM classifier, for optimal kernel parameters Kernel Type% Correct 1 Scale 3 Scales Radial Basis Histogram Intersection Chi Square

Classification results – confusion matrix Confusion matrix of random 2000 test images (2007 labels) 91.95% correct

Submission to ImageClef 2009 medical annotation task o One run submitted o Use the same classifier for the 4 label sets (2005,2006,2007,2008) o Ignore IRMA code hierarchy o Don’t use wildcards Run & error score SUM TAUbiomed

Conclusion & future work o Using visual words with simple features and dense sampling is efficient and accurate in general x-ray annotation o We are applying the system to pathology classifications of chest x-rays, together with Sheba Medical Center Healthy Enlarged heart Lung filtrate Left+right effusion

Thank you.