Multimedia Search and Retrieval Presented by: Reza Aghaee For Multimedia Course(CMPT820) Simon Fraser University March.2005 Shih-Fu Chang, Qian Huang, Thomas Huang, Atul Puri, and Behzad Shahraray Published as a chapter in Advances in Multimedia: Systems, Standards, and Networks, A. Puri and T. Chen (eds.). New York: Marcel Dekker, 1999.

Agenda Introduction Video Segmentation, Indexing and Browsing Object-based Spatio-Temporal Search Semantic-Level Content Classification and Filtering Multimedia Meta Search Engines Conclusions 2/24

Introduction Applications: WEB Large-scale Multimedia Search Engines Media Asset Management Systems Audio-Visual Broadcast Servers Personal Media Servers for Consumers 3/24

Video Segmentation, Indexing and Browsing Hierarchical Segmenting –smaller retrievable data units Hierarchical Grouping –larger yet meaningful categories Layers of abstraction –commercials, news stories, news introductions, news summaries of the day 4/24

Video Segmentation, Indexing and Browsing Low-level segmentation of video streams –Streams are segmented into shots, clips and key frames. –They Do not correspond to the real semantic structure –Large amount of low-level structures, hence, browsing inefficiency 5/24

Video Segmentation, Indexing and Browsing Semantic Segmentation of news programs 6/24

Video Segmentation, Indexing and Browsing Relationship among semantic structures 7/24

Video Segmentation, Indexing and Browsing Representation & Browsing Tools –Time lined presentation 8/24

Video Segmentation, Indexing and Browsing Representation & Browsing Tools -Visual Pres. For stories about E1 Nino 9/24

Video Segmentation, Indexing and Browsing Representation & Browsing Tools -Visual Pres. For stories about E1 Nino 10/24

Object-based Spatio-Temporal Search & Filtering Query by: –Example Meaningful real world objects Low-level image regions with uniform features –Feature Color, Texture, Shape, Motion, Spatio-temporal structure of image regions –Sketches Users directly draw visual sketches 11/24

Object-based Spatio-Temporal Search & Filtering Object-oriented search by feature and sketches (a) & (c) are sketches by the user (b) & (d) are returned as results 12/24

Object-based Spatio-Temporal Search & Filtering VideoQ search system –Video decomposed into shots –Shot separation achieved by scene change detection –Salient video regions and objects extracted –Temporal attributes of regions are indexed 13/24

Object-based Spatio-Temporal Search & Filtering Query processing architecture 14/24

Object-based Spatio-Temporal Search & Filtering Interface of AMOS semantic object search engine 15/24

Semantic-Level Content Classification and Filtering Idea is mapping images or videos to meaningful classes Content modeling using probabilistic graphic models –Multiject (multimedia object) Has a label Summarizes the time sequences in from a probabilities, P( label | sequences) 16/24

Semantic-Level Content Classification and Filtering –Multiject categories Sites, Objects, Events –Multiject Lifetime Duration of multimedia input used to determine its probability –Multinet (multiject network) Represents probabilistic dependencies between multijects 17/24

Semantic-Level Content Classification and Filtering A multinet describes probabilistic dependencies between multijects 18/24

Semantic-Level Content Classification and Filtering Indexing Multimedia with semantic templates (STs) –Use a set of successful queries instead of a single one –There could be audio or video templates or both Semantic Visual Templates 19/24

Semantic-Level Content Classification and Filtering Components in development of STs –Generation Used to generate STs for each semantic concept –Metric Used to measure the fitness (similarity) of each ST –Applications Used to develop a library of semantic concepts to facilitate video query 20/24

Semantic-Level Content Classification and Filtering Semantic Template Development - Slalom 21/24

Meta Search Engines Gateways Linking users transparently to multiple search engines 22/24

Meta Search Engines Basic Components –Query dispatcher Selects target search engines by performance scores –Query translator Translate query to a suitable script for the target –Display interface Merges the results of each engine using performance scores 23/24

Conclusions Semantic segmentation instead of low-level segmentation Methods of semantic segmentation Object-based semantic searches Probabilistic models and template-based searches Meta search engines architecture 24/24