RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES Mingming Lu, Qiyu Zhang, Wei-Hung Cheng, Cheng-Chang Lu Department of Computer Science Kent State University

September, 2009Kent State University2 Data Mining and Knowledge Management Processing multimedia objects Defining and extracting features Feature dimension reduction Multimedia data retrieval Knowledge representation and management

September, 2009Kent State University3 Current Tasks Off-line data training –Segment images – batch mode –Find region of interest (ROI) –Interface with feature extraction and analysis –Feature domain processing

September, 2009Kent State University4 Current Tasks (cont.) Users Interfaces –Reading user-input images –Segmentation –Find ROI –Feature extraction of ROI –Compare with trained data in repository –Return data (images) satisfying certain criteria

September, 2009Kent State University5 Data Training Image & Feature Data Repository SegmentationFinding ROI Interface Feature Extract Dimension Reduction Sending Images for ProcessingStore Feature Data back Image DomainFeature Domain

September, 2009Kent State University6 Image Domain Procesisng Segmentation – Color VQ, Texture based image segmentation Find ROI –ROI occupies large area –ROI locates near the image center –ROI contains homogenous texture

September, 2009Kent State University7 Color-Texture Segmentation Applications Identify Regions of Interest (ROI) in a scene Image classification Image annotation Object based image and video coding

September, 2009Kent State University8 Color-Texture Segmentation Current Limitations Many existing techniques work well on homogeneous color regions, while natural scenes are rich in color and texture. Many texture segmentation algorithms require the estimation of texture model parameters, which is a difficult problem and often requires a good homogeneous region for robust estimation.

September, 2009Kent State University9 Color-Texture Segmentation Advantage of Color VQ and Texture based segmentation Does not attempt to estimate a specific model for a texture region. Tests for the homogeneity of a given color- texture pattern, which is computationally more feasible than estimation of model parameters.

September, 2009Kent State University10 Color-Texture Segmentation Two-Step Process Color Quantization –Performed in the color space without consideration of spatial distribution of colors. –Label each pixel with a quantized color to form a class-map. Spatial Segmentation –Performed on the class-map

September, 2009Kent State University11 Color-Texture Segmentation Color Quantization Use Peer Group Filtering As a result, coarse quantization can be obtained while preserving the color information in the original images. Usually colors are needed in the images of natural scenes.

September, 2009Kent State University12 Color-Texture Segmentation Criteria for Good Segmentation

September, 2009Kent State University13 Color-Texture Segmentation - A Criterion for Good Segmentation When the color classes are more separated from each other, J is getting larger. If all color classes are uniformly distributed over the entire image, J tends to be small.

September, 2009Kent State University14 Color-Texture Segmentation A Criterion for Good Segmentation Now let us recalculate J over each segmented region instead of the entire class-map and define the average by A segmentation which can minimize J is considered a good segmentation.

September, 2009Kent State University15 Color-Texture Segmentation - Spatial Segmentation Seed Determination Seed Growing Region Merge

September, 2009Kent State University16 Color-Texture Segmentation - Spatial Segmentation

September, 2009Kent State University17 ROI Determination Find ROI – Mechanism –Pixel closer to the center contributes more weight to the region it belongs to. –Region with more pixels tends to get higher weight

September, 2009Kent State University18 Results of Image Domain Processing Results of Color Quantization Results of Finding ROI

Results of Image Domain Processing March, 2004Kent State University19 V1 = 500, V2 = 1, V3 = 0.5AutoV1 = 500, V2 = 1, V3 = 0.5

Results of Image Domain Processing March, 2004Kent State University20 V1 = 500, V2 = 1, V3 = 0.5Auto

March, 2004Kent State University21

September, 2009Kent State University22 Interface with Feature Domain Find the rectangle circumscribing the ROI Store its coordinate information into to a temporary file for feature domain’s use.

September, 2009Kent State University23 Feature Domain(Overview) Two Stages: –Feature Extraction –Dimension Reduction (DR) Image & Feature Data Repository Interface Store Feature Data back Image Domain Feature Extract Dimension Reduction Feature Domain

September, 2009Kent State University24 Implementations Acquire ROI information from the image domain Extract features based on Gabor Filter and color histogram on HSV space Integrate two feature spaces Reduce the high feature dimensions to a very low number

September, 2009Kent State University25 Implementations (cont.) Calculate the similarity measurement between the query object and the objects in the image repository Search the similar images in the repository based on similarity index Output the corresponding retrieval images Knowledge extraction

September, 2009Kent State University26 Feature Extraction Algorithm Gabor Filter Feature –One of the most important wavelets with multi- scale and multi-resolution –Mainly reflect texture information Color histogram on HSV space –Provide color features

September, 2009Kent State University27 Gabor Filter Concept A complete but non-orthogonal basis wavelet set A significant aspect: localized frequency description – composed of space information

September, 2009Kent State University28 Gabor(cont.) A two dimensional Gabor function g(x, y) and its Fourier transform G(u, v) can be written as:

September, 2009Kent State University29 Gabor(cont.) Let g(x, y) be the mother Gabor wavelet, then this self-similar filter dictionary can be obtained by appropriate dilations and rotations of g(x, y) through the generating function

September, 2009Kent State University30 Color Histogram in HSV Space HSV color space includes –Hue (H) –Saturation (S) –Value (V or Lightness) Only consider Hue and saturation information, since the lightness of pictures is very sensitive to the surrounding conditions.

September, 2009Kent State University31 HSV space Figure

September, 2009Kent State University32 HSV space bands Design bands in the HSV space –8 hue bands –4 saturation bands, –Total 32 sub-spaces Compute color histogram feature in each sub-space to form 32 feature dimensions eventually

September, 2009Kent State University33 Feature Integration Normalize both Gabor filter and HSV color histogram features Set a weight factor to balance two feature spaces. Usually Gabor filter features will have the bigger weight value.

September, 2009Kent State University34 DR Algorithm Disadvantages in the high dimension space –The computational complexity arise sharply –The database indexing becomes difficult Principal Component Analysis (PCA) –PCA seeks to reduce the dimension of the data by finding a few orthogonal linear combinations (Principal Component “PC”)

September, 2009Kent State University35 DR implementation Original feature dimensions –Gabor filter features: 6*5*2 = 60 –HSV color histogram features: 4*8 = 32 –Total dimensions: 92 Feature dimensions after DR –10 ~15 dimensions

September, 2009Kent State University36 Simulation Results in the Feature Domain We randomly select 11 query pictures as the test samples in this report. At each query time, at most 14 retrieval pictures are retrieved. The minimum square error method is served as the similarity measurement. The value in the tables as below means the positive pictures out of the 14 retrieval pictures.

September, 2009Kent State University37 Performance between different feature extraction techniques the integration of Gabor Filter and HSV color Histogram gains the better performance. See pictures in detail. Click hereClick here Query pic# Gabor HSV Integrated

September, 2009Kent State University38 Performance between with and without DR applied The performance after DR applied slightly degrades on average in comparison to the results before DR takes on stage See pictures in detail. Click hereClick here Query pic# Integrated DR

September, 2009Kent State University39 More Simulations Performance between different weight used Performance between different dimensions retained after DRPerformance between different dimensions retained after DR

September, 2009Kent State University40 Final Integration Results Simulation results when both the image domain and the feature domain are used See the detail pictures, Click hereClick here

September, 2009Kent State University41 Integration UAV media capture and analysis WWW based media analysis Vehicle based media capture and analysis

September, 2009Kent State University42 Future Research Extended to video objects Object based video coding Non-object based video coding Video indexing Knowledge extraction and management

September, 2009Kent State University43 Future Research Data Fusion Multimodality medical imaging CT – Structural information PET – Functional information Fusion Knowledge management