Content-Based Image Retrieval - Approaches and Trends of the New Age Ritendra Datta, Jia Li, and James Z. Wang The Pennsylvania State University MIR2005

INTRODUCTION 為什麼image無法處理的像text一樣好 Text is man’s creation, images are a mere replica of what man has seen Interpretation of what we see is hard to characterize visual similarity != semantic similarity CBIR has grown tremendously after 2000, not just in terms of size, but also in the number of new directions explored

INTRODUCTION

INTRODUCTION The theoretical foundation behind how we humans interpret images is still an open problem A brief scanning of about 300 relevant papers published in the last five years revealed that less than 20% were concerned with applications or real-world systems

CBIR領域研究方向 Feature Extraction Approaches to Retrieval Annotation and Concept Detection Relevance Feedback and Learning Hardware and Interface Support

Feature Extraction 如何抽 Color Feature “An Efficient Color Representation for Image Retrieval” (比傳統histograms好) “Multiresolution Histograms and Their Use for Recognition” (用在textured image) “Image retrieval using color histograms generated by Gauss mixture vector quantization” (利用GMVQ抽color histogram)

Feature Extraction Color + Texture 抽取 Shape “Wavelet-Based Texture Retrieval Using Generalized Gaussian Density and Kullback-Leibler Distance” Shape “Shape Matching and Object Recognition Using Shape Contexts” (is fairly compact yet robust to a number of geometric transformations)

Feature Extraction Segmentation “Normalized Cuts and Image Segmentation” (最重要的方向之一) “Blobworld: Image Segmentation Using Expectation-maximization and Its Application to Image Querying” (我之前用過的方法) “Segmentation of Brain MR Images Through a Hidden Markov Random Field Model and the Expectation-Maximization Algorithm” (處理medical imaging)

Feature Extraction 線條相似度 如何選擇feature “Image retrieval using wavelet-based salient points” 如何選擇feature Application-specific feature sets (最直觀的) “SIMPLIcity:Semantics-Sensitive Integrated Matching for Picture Libraries” (semantics-sensitive feature selection) “Feature Selection for SVMs” (用classifier)

Approaches to Retrieval Region based image retrieval “A Scalable Integrated Region-Based Image Retrieval System” region-based querying (BlobWorld) Vector quantization (VQ) on image blocks “Keyblock: An Approach for Content-based Image Retrieval” (generate codebooks for representation and retrieval, taking inspiration from data compression and text-based strategies)

Approaches to Retrieval Windowed search “Object-Based Image Retrieval Using the Statistical Structure of Images” (more effective than methods based on inaccurate segmentation) Anchoring-based image retrieval “A Study of Image Retrieval by Anchoring” (Anchoring is based on the idea of finding a set of representative “anchor” images and deciding semantic proximity between an arbitrary image pair in terms of their similarity to these anchors)

Approaches to Retrieval Probabilistic frameworks for image retrieval “A Probabilistic Architecture for Content-based Image Retrieval”

Annotation and Concept Detection Supervised classification “Image Classification for Content-Based Indexing” (involving simple concepts such as city, landscape, sunset,and forest, have been achieved with high accuracy) Translation approach “Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary” (我們在clef 2004就是follow這方法)

Annotation and Concept Detection 為何如此困難 We humans segment objects better than machines, having learned to associate over a long period of time, through multiple viewpoints, and literally through a “streaming video” at all times The association of words and blobs become truly meaningful only when blobs isolate objects well

Relevance Feedback and Learning “Relevance Feedback in Image Retrieval: A Comprehensive Review” Problems One problem with RF is that after every round of user interaction, usually the top results with respect to the query have to be recomputed Another issue is the user’s patience in supporting multi-round feedbacks

REAL-WORLD REQUIREMENTS Performance Semantic learning Volume of Data Concurrent Usage Heterogeneity Multi-modal features User-interface Operating Speed System Evaluation

CURRENT RESEARCH TRENDS Journals IEEE T. Pattern Analysis and Machine Intelligence (PAMI) IEEE T. Image Processing (TIP) IEEE T. Circuits and Systems for Video Technology (CSVT) IEEE T. Multimedia (TOM) J. Machine Learning Research (JMLR) International J. Computer Vision (IJCV)

CURRENT RESEARCH TRENDS Pattern Recognition Letters (PRL) ACM Computing Surveys (SURV) Conferences IEEE Computer Vision and Pattern Recognition (CVPR) International Conference on Computer Vision (ICCV) European Conference on Computer Vision (ECCV) IEEE International Conference on Image Processing (ICIP)

CURRENT RESEARCH TRENDS ACM Multimedia (MM) ACM SIG Information Retrieval (IR) ACM Human Factors in Computing Systems (CHI)

CONCLUSIONS We have presented a brief survey on work related to the young and exciting fields of content-based image retrieval and automated image annotation, spanning 120 publications in the current decade We have laid out some guidelines for building practical, real-world systems