Color-based Diagnosis: Clinical Images Research Project Funded In Part by NIH Yue (Iris) Cheng, Dr. Scott E Umbaugh @ Computer Vision and Image Processing Research Lab Electrical and Computer Engineering Department Southern Illinois University Edwardsville E-mail: cheng@westar.com https://www.ee.siue.edu/CVIPtools

Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE Overview Skin tumors can be either malignant or benign Clinically difficult to differentiate the early stage of malignant melanoma and benign tumors due to the similarity in appearance Proper identification and classification of malignant melanoma is considered as the top priority because of cost function Classification of skin tumors using computer imaging and pattern recognition Previous texture feature algorithm successfully differentiate the deadly melanoma and benign tumor seborrhea kurtosis Relative color feature algorithm is explored in this research for differentiate melanoma and benign tumors, dysplastic nevi and nevus Successfully classify 86% of malignant melanoma using relative color features, compared to the clinical accuracy by dermatologists in detection of melanoma of approximately 75% 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE Materials and Tools Image database Original tumor images 512x512 24-bit color images digitized from 35mm color photographic slides and photographs 160 melanoma, 42 dysplastic, and 80 nevus skin tumor images Border images Binary images drawn manually and reviewed by the dermatologist for accuracy Software CVIPtools Computer vision and image processing tools developed at our research lab Partek Statistical analysis tools 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE CVIPtools 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE Method Design Creation of relative color images Segmentation and morphological filtering Relative color feature extraction Design of tumor feature space and object feature space Establishing statistical models from relative color features 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Create Relative Color Skin Tumor Images Purpose to equalize any variations caused by lighting, photography/printing or digitization process to equalize variations in normal skin color between individuals the human visual system works on a relative color system Algorithm Mask out non-skin part in the image to calculate the normal skin color Separate tumor from the image Remove the skin color from the tumor to get a relative color skin tumor image CVIPtools functions were used to create relative color skin tumor images 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Calculate Skin Color Original Noisy Skin Tumor Image Non-skin Algorithm Calculate Mask out tumor Skin Tumor Image W/O Noise Average R, G, B Value of Skin Skin-Only Image 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Tumor Image Original Noisy Skin Tumor Image Tumor Border Image AND Original Noisy Skin Tumor Image Border Image Tumor 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Relative Color Tumor Image SUBTRACT Tumor Image Average R, G, B Value of Skin Relative Color Image of the Tumor 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Segmentation and Morphological Filtering Image segmentation was used to find regions that represent objects or meaningful parts of objects Morphological filtering was used to reduce the number of objects in the segmented image Easy to use CVIPtools for experimenting and analysis 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Relative Color Feature Extraction Necessary to simplify the raw image data into higher level, meaningful information Feature vectors are a standard technique for classifying objects, where each object is defined by a set of attributes in a feature space. Totally 17 color features and binary features were extracted using CVIPtools The three largest objects, based on the binary feature ‘area’, were used in feature extraction Histogram features, that is, color features, were extracted in each color band from relative color image objects 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE 17 Features Binary features Area Thinness Histogram features in R, G, B bands Mean Standard deviation Skewness Energy Entropy 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE 17 Features (Cont.) 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Design Two Feature Spaces Tumor feature space consists of 277 feature vectors correspond to 277 skin tumor images. each feature vector has 51 feature elements, which are the total of 17 features of each three largest objects within the same tumor. Object feature space had 842 feature vectors corresponding to 842 image objects each feature vector has 17 feature elements, which were the binary features and color features stated as above 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Establishing Statistical Models Two feature spaces serve as two data models in order to maximize the possibility of success Two classification models, Discriminant Analysis and Multi-layer Perceptron, were developed for both data models The training and test paradigm is used in statistical analysis to report unbiased results of a particular algorithm due to small size of data set, 282 images, we used the leave x out method, with both one and ten for x Partek software was used to analyze the data representing the features to develop a model or rules for classifying the tumors 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Quadratic Discriminant Analysis A statistical pattern recognition technique based on Bayesian theory, which classifies data based on the distribution of measurement data into predefined classes Normalization the feature data as preprocessing performed to maximize the potential of the features to separate classes and satisfy the requirement of the modeling tool such as Quadratic discriminant analysis for a Bayesian distribution of the input data Variable selection was used to choose dominant features. 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Multi-Layer Perceptron A feed forward neural network neural networks modeled after the nervous system in biological systems, based on the processing element the neuron widely used for pattern classification, since they learn how to transform a given data into a desired output. Principal Component Analysis (PCA) as preprocessing a popular multivariate technique, is to reduce dimensionality by extracting the smallest number components that account for most of the variation in the original multivariate data and to summarize the data with little loss of information the dispersion matrix selected for PCA in this project is correlation 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Multi-Layer Perceptron (Cont.) Creation, training and testing of neural networks Creation a neural network involves selection of hidden and output neuron types and a random number generation. Four output neuron types – Softmax, Gaussian, Linear and sigmoid Three hidden neuron types – Sigmoid, Gaussian and Linear Scaled Conjugate Gradient algorithm is used for learning in this project. Automated and independent of user parameters Avoids time consuming Stopping criteria, sum-squared error, is selected to determine after how many iterations the training should be stopped The trained data is then tested on itself first to examine how far the neural network is able to classify the objects correctly. Leave x partition out method is used for testing the algorithm 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Experiments and Analysis in Tumor Feature Space Discriminant Analysis 24 features selected for leave ten out method Histogram Features Mean STD Skewness Energy Entropy R G B Object 1 X Object 2 Object 3 10 features selected for leave one out method Histogram Features Mean STD Skewness Energy Entropy R G B Object1 X Object 2 Object 3 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Experiments and Analysis in Tumor Feature Space (Cont.) Discriminant Analysis (Cont.) 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Experiments and Analysis in Tumor Feature Space (Cont.) Multi-layer Perceptron Best features, being in the first three components of the PCA projection data, were used Success percentages of melanoma as high as 77% and nevus is as high as 68% 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Experiments and Analysis in Object Feature Space Discriminant Analysis 8, 9, 11 and 12 significant features were selected respectively for leave one out method Number of Histogram Features Area Mean STD Skewness Energy Entropy R G B 8 X 9 11 12 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Experiments and Analysis in Object Feature Space (Cont.) Discriminant Analysis (Cont.) Yield consistent results in classifying melanoma from other skin tumor with above 80% success rate 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Experiments and Analysis in Object Feature Space (Cont.) Multi-layer Perceptron (MLP) 5 out of 12 hidden-output layer neuron combinations gave better classification results Leave one out method Yield success percentage as high as 86% for classifying melanoma. MLP is more consistent in classifying melanoma as well as nevus 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE Conclusion Multi-Layer perceptron (MLP) with feature data preprocessed by Principal Component Analysis (PCA) gave better classification results for melonoma than Discriminant Analysis (DA) The best overall successful rate of 78%, of which percentage correct of melanoma is 86%, nevus is 62% and dysplastic is 56%. The best classification results are achieved with sigmoid used as the hidden and output layer neuron type for the MLP with PCA on Object Feature Space. The three largest tumor objects are representative for the whole skin tumor. 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE Conclusion (Cont.) However the small percentage of melanoma misclassification as well as the relatively low success rate for nevus and dysplastic nevi suggests that we may not have the complete data set for the experiments. In order to achieve better classification results, future experiments Needs more complete skin tumor image database. Should combine texture and color methods to get better results Will include dermoscopy images 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE Acknowledgement Dr. Scott E Umbaugh, SIUE Mr. Ragavendar Swamisai Ms. Subhashini K. Srinivasan Ms. Saritha Teegala Dr. William V. Stoecker, Dermatologist, UMR 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE

Thank You! Yue (Iris) Cheng Graduate Student @ Computer Vision and Image Processing Research Lab Electrical and Computer Engineering Department Southern Illinois University Edwardsville E-mail: cheng@westar.com https://www.ee.siue.edu/CVIPtools 07/13/2005 Computer Vision and Image Processing Research Lab @ ECE Dept., SIUE