Diagnostic Accuracy of Content Based Dermatoscopic Image Retrieval with Deep Classification Features Tschandl P1,2, Argenziano G3, Razmara M4, Yap J4 1 - Department of Dermatology, Medical University of Vienna, Vienna, Austria 2 - School of Computing Science, Simon Fraser University, Burnaby, Canada 3 - Department of Dermatology, University of Campania, Naples, Italy 4 - MetaOptima Technology Inc. British Journal of Dermatology. DOI: 10.111/bjd.17189

Lead Researcher Philipp Tschandl

Introduction What’s already known? Convolutional Neural Networks (CNN) can detect skin cancer on digital images comparable to dermatologists in experimental settings. CNNs may be difficult to implement in practice as they commonly output numerical disease probabilities only. Deep features in intermediate stages of a CNN correspond to high-level visual properties on an image.

Objective To compare the diagnostic accuracy obtained by using CBIR (content-based image retrieval) retrieving visually similar dermatoscopic images against conventional predictions made by a neural network.

Methods (1) Automated retrieval of similar images from a database can show visually similar cases. Diagnostic probabilities were inferred from the frequency of retrieved diagnoses. CBIR-prediction: 100% Nevus CBIR-prediction: 50% SCC, 50% BCC CBIR-prediction: 100% Melanoma

Methods (2) A ResNet-50 network architecture was trained for classification on one of three different datasets at a time. Comparing the feature vectors of two images with cosine-similarity was used to measure visual „similarity“ between them.

Results (1) Images with the same diagnosis have a higher similarity value than „others“ grouped together.

Results (2) Only a few visually similar cases need to be retrieved to obtain the same accuracy as softmax-based prediction probabilities. In an 8-class problem (PRIV dataset) CBIR-based predictions could only approximate softmax-prediction based accuracy.

Results (3) CBIR-based performance is highest when test-images (rows) come from the same dataset as the retrieval images (columns). Lowest row: CBIR (black) is able to surpass softmax (red) based performance when the network „knows“ only three classes, but needs to predict eight classes when it has access to the more comprehensive retrieval data (right-most column). Softmax based predictions finetuned on the target data (blue) always have the highest performance.

Discussion (1) Similarity-based retrieval from more comprehensive sets with more diagnoses can enable networks to predict “unseen” diagnoses. This may be interesting if medical centres cannot retrain a neural network on their own images – but still, want to use them.

Discussion (2) CBIR based predictions look reasonably accurate, but efficient human-computer-interaction for this method needs to be determined. Accuracy may differ if a user can choose which retrieval cases should be deemed relevant.

Discussion (3) Case retrieval, rather than providing diagnosis probabilities could aid a decision process rather than take it away.

Conclusions What does this study add? Content-based image retrieval (CBIR) based on deep features can retrieve visually similar dermatoscopic images. Retrieving only 16 similar images can achieve the same accuracy as a CNN classifier. CBIR can enable a CNN to recognise unknown disease classes in new datasets.

Research Team Philipp Tschandl Jordan Yap Majid Razmara Guiseppe Argenziano

Call for correspondence Why not join the debate on this article through our correspondence section? Rapid responses should not exceed 350 words, four references and one figure Further details can be found here