Visual Information Retrieval Introduction

Visual Information Retrieval Information retrieval, image/video analysis and processing, pattern recognition and computer vision, visual data modeling and representation, multimedia database organization, multidimensional indexing, psychological modeling of user behavior, man-machine interaction and data visualization

Visual Information Retrieval Types of associated information content-independent metadata (CIM) format, author's name, date, location, ownership content-dependent metadata (CDepM) low-level features concerned with perceptual facts: color, texture, shape, spatial relationship, motion content-descriptive metadata (CDesM) high-level content semantics: cloud

Visual Information Retrieval First-generation visual information retrieval systems CIM by alphanumeric strings, CDepM and CDesM by keywords or scripts

Visual Information Retrieval find images of paintings by Chagall with a blue background Select IMAGE# from PAINTINGS where PAINTER = "Chagall" and BACKGROUND = "blue" find images of paintings by Chagall with a girl in red dress and a blue background full text retrieval

Visual Information Retrieval find images of paintings depicting similar figures in similar positions it is difficult for text to capture the perceptual saliency of some visual features text is not well suited for modeling perceptual similarity perception is mainly subjective, so is its text descriptions

Visual Information Retrieval New-generation visual information retrieval systems retrieval not only by concepts but also by perception of visual contents objective measurements of visual contents and appropriate similarity models automatically extract features from raw data by image processing, pattern recognition, speech analysis and computer vision techniques

Visual Information Retrieval Image retrieval by perceptual features for each image in the database, a set of features (model parameters) are precomputed to query the image database express the query through visual examples select features and ranges of features choose a similarity measure compute similarity degrees, ranking, relevance feedback

Visual Information Retrieval

Visual Information Retrieval system architecture extraction of perceptual features (CDepM) extraction of high-level semantics (CDesM) from low- level features manual annotation of CIM and CDesM index structure graphical query tool retrieval engine visualization tool relevance feedback mechanism

Visual Information Retrieval Video retrieval special characteristics frames are linked together using editing effects cut, fade, dissolve content of the frames characters, story changes in color, texture, shape and position (camera or object) in multiple frames techniques to obtain video streams different types of video commercial, news, movies, sport

Visual Information Retrieval by structure frame: basic unit of information shot: elementary segment of video with perceptual continuity clip: set of frames with some semantic meaning scene: consecutive shots with simultaneous space, time and action episode: specific sequence of shot types such as a news episode

Visual Information Retrieval by content perceptual properties, motion and type of an object situations between objects motion of camera semantics of shots by color- or motion-induced sensations semantics of scenes stories audio properties: dialogue, music or storytelling textual information: caption or text recognized from video

Visual Information Retrieval system architecture extraction of shots and the associated semantics key-frames or mosaics extraction of scenes and stories manual annotation tool browsing/visualization tool video summarization graphical query tool index structure retrieval engine

Visual Information Retrieval 3D image and video retrieval WWW visual information searching efficiency has to be emphasized due to limited network bandwidth operate in compressed domain visual summarization visualization at different levels of resolution

Visual Information Retrieval Research directions tools for automatic extraction of low-level features tools for automatic extraction of high-level semantics models for representing visual content at several abstraction levels effective indexing effective database models

Visual Information Retrieval visual interfaces allow querying and browsing allow querying by text and visual information similarity models fit human similarity judgment psychological similarity models Web search content + references 3D image and video retrieval