Imagery-based Digital Collection Retrieval on Web Using Compact Perception Features Ying Dai Faculty of software and information science, Iwate Pref. University 2018/11/28

Introduction More and more generation of image information led to The emergency of automatic index; The emergency of eliminating the semantic gap between the user and the retrieval system; The emergency of flexible and intuitive retrieval 2018/11/28

Architecture of image retrieval by user intuitive queries (1) Following the conclusion by B. E. Rogowize [1] Human observers are very systematic in the evaluation of image similarity, following Semantic Color Structural characteristics Two most fundamental dimensions describing semantic of images Human vs. non-human Natural vs. man-made Two most fundamental dimensions for humans in judging image similarity: Human vs. non-human axis: running from the less human-like images of sunsets and clouds, scenes with animals, scenes with small images of humans, to images featuring large full-face portraits of humans. Natural vs. man-made axis: running from the rock and flower images, nature scenes, nature scenes with man-made objects, to man-made objects with nature, to the building images. [1] B. E. Rogowitz, et al. “Perceptual image similarity experiments”, Proc. Of SPIE, 1998. 2018/11/28

Architecture of image retrieval by user intuitive queries (2) Architecture of image retrieval which cope with users’ intuitive queries The system include 5 interfaces. Category interface perceives the user’s queries regarding semantic to images. Perception word, impression word, and sample interface perceive the users’ queries regarding color and composition to images. The feedback interface perceives user’s individual similarity criteria. Category database indexes the images by the semantic category, features set DB indexes images by the computation features. The retrieval modules map the users’ queries to the category or features set database, and retrieve the corresponding images. The adjusting module map the individual criteria to the decision thresholds, storage the adjusted thresholds, and deliver them to the retrieval modules. 2018/11/28

Semantic category of images Nature with man-made objects man-made objects nature scenes Without beings With beings With full-face portraits Without beings With beings With full-face portraits building, interior, etc. indoor with humans, etc. Wild Flower, mountain, sunset, etc. Wild animal, picnic, etc. Without beings with beings Images can be categorized in the tree structure based on the mentioned conclusion Flower arrangement, etc. Amusement grounds, etc. 2018/11/28

Cues for image categorization Annotation Date Position Definition by user Computation feature set position: Japan’s geographical data Annotation: A image of Japanese garden definition: garden Feature set: green, grey nature scene 2018/11/28

Extraction of feature set Unit Nu. Signification Property 3 Means of row sums of SGLD matrices of H, S, V components Hue and tone distribution Deviations of row sums of SGLD matrices 5 Eigen values of SGLD matrix of H components The number of hues Means of row sums of Eigen SGLD matrix of H components The principal hues 60 Textural feature parameters of second difference SGLD matrices Composition-based similarity classification Here, we use the eigen and difference SGLD matrices to extract the low-level features of images. For the details, the paper presented in IEEE SMC2004 can be referred. [2] Ying Dai, et al. “Visual perception-based structure analysis of images for digital collection retrieval”, IEEE Proc. Of SMC 2004, CD-ROM, pp. 1104-1111, Oct., 2004, Netherlands. 2018/11/28

Extraction of objects in images by using Condition for extracting: Original image 2018/11/28

application of individual criteria in mapping query to features (1) Objective perception-based color mapping For the query such as complex hue, red, an so on, By adjusting the threshold set TO regarding n and u, the individual user’s perceptual similarity criteria to the color can be reflected. 2018/11/28

application of individual criteria in mapping query to features (2) Subjective perception-based mapping 2018/11/28

application of individual criteria in mapping query to features (3) Composition similarity mapping 　　　are used to measure the similarity of images in composition by Euclidean distance 2018/11/28

Some image retrieval results (1) Query: retrieving images in the category of papers, fabrics & woods, which are similar to a reference image in structure. A reference Retrieved results 2018/11/28

Some image retrieval results (2) Query: retrieving images in the category of paints & pastels, which have multi-hues, appearing calm. Retrieved results 2018/11/28

Some image retrieval results (6) Query: retrieving images in the category of flowers, which are similar to a reference image in overall color and structure. A reference Retrieved results ( 6 best matches) 2018/11/28

Satisfaction-based evaluation Evaluation items: 1) The initial ranking is satisfy; 2) The modified ranking is satisfy; 3) number of images which are near to your imagery for the first ranking of top 10; 4) number of images which are near to your imagery for the modified ranking of top 10; a), b), c), d) are the average values of positive answers of 1), 2), 3), 4) Number of images Image database Number　of images as query a) b) c) d) 200 Vol. 2 10 77% 86% 6 7 Vol. 9 83% 90% Vol. 10 80% 8 Vol. 20 2018/11/28

Future work & Reference Improving the Automatic indexing of semantic category User-based evaluation Flexibility Satisfaction Ease to use Reference [1] B. E. Rogowitz, et al. “Perceptual image similarity experiments”, Proc. Of SPIE, 1998. 2018/11/28