Multimedia Search and Retrieval: New Concepts, System Implementation, and Application Qian Huang, Atul Puri, Zhu Liu IEEE TRANSACTION ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, AUGUEST 2000

Outline Introduction Media Types and Their Properties Multimedia Data Abstraction and Retrieval System Architecture System Implementation An Application: Multimedia Broadcast News Summary

Introduction MPEG-7 focuses on the standardization of a common interface for describing multimedia materials,supporting pull and push information. Our proposed multimedia information retrieval system cover system architecture, data storage and representation, database and its remote access, and human-machine interfaces. The concept of physical and logical layers and events.

Media Types and Their Properties Audio Audio events can be defined as continuous audio segments that have some distinct acoustic characteristics Visual Text

Multimedia Data Abstraction and Retrieval A physical layer addresses the media events that are time continuous and better described using the specific characteristics of a media signal. A logical layer addresses multimedia events that present integrated behavior of other events across media.

Multimedia Data Abstraction and Retrieval(cont.) Physical Program and Physical Event Physical program a continuous stream of multimedia data that begins and ends due to the physical scene edits and cuts decided by human operators Physical event A continuous segment of multimedia data, representing a coherent and meaningful event within a physical program

Multimedia Data Abstraction and Retrieval(cont.) Logical Event A virtual view of the multimedia data constructed from a set of physical programs or events, or recursively from other physical or logical events. Application dependent Semantic Content Structure Conceptual and virtual Provide a seamless interface to user Table of Content (ToC)

Relationship among different event layers

System Architecture

System Implementation Server Implementation Content Hierarchy Extraction Layered Content Storage Store lower level information in a file system and higher level information in standard database systems When raw data is stored in a file system, different events in the data stream can be accessed using time based indexing Database Content Access Employ JDBC to provide connectivity to our database for query and update

Multimedia data storage in combined file system and database

Example of event indexing from database to a file

Database access

System Implementation(cont.) Client Browser Implementation Java applet-based interface Platform independence support for a variety of coded media Remote Database Query RMI (Remote Method Invocation) Browsing Interface Table of Content: a conceptual content structure TimeLine: a content structure in time Event Presentation: presentation and playback Content Playback with a JMF Player

Remote database access

Browser interface for a news program with both ToC and timeline representations of the semantic structure within the program

An Application: Multimedia Broadcast News Server End Content Hierarchy Extraction Broadcast news data is well structured. Lower levels – extract primitive physical events in each individual media stream. Higher semantic level – merge different events into semantically meaningful logical events. ex: Headline stories and news summaries.

Content hierarchy of broadcast news programs

Content Storage and Access In the file system, for each physical broadcast news program, store the timetables for audio, video,and text,individually. The tables in the database system will index into these timetables for the actual retrieval and data access. Physical events in each media are recorded in separate database tables at the physical event layer.

Relationship between logical event and physical event objects

Database design in broadcast news application

Client End Remote Database Query The physical program ID is uniquely determined by the channel and the date of the program. Content Presentation Content Structure Presentation ToC and TimeLine Static(single click) and playback(double click) Story Presentation News Summary Presentation Streaming Playback

Browser interface for a news program with both ToC and timeline representations of the semantic structure within the program

Visual representation for story “ El Nino ”

Representation for news summary of the day

Playback interface with synchronized audio/video/text

Summary We have introduced a new layered approach to multimedia content representation and storage for search and retrieval. The concepts are implemented in a client/server multimedia information retrieval system designed for broadcast news search and retrieval.