1. Multimedia Search and Retrieval: New Concepts, System Implementation, and Application Qian Huang, Atul Puri, Zhu Liu IEEE TRANSACTION ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, AUGUEST 2000

2. Outline Introduction Media Types and Their Properties Multimedia Data Abstraction and Retrieval System Architecture System Implementation An Application: Multimedia Broadcast News Summary

3. Introduction MPEG-7 focuses on the standardization of a common interface for describing multimedia materials,supporting pull and push information. Our proposed multimedia information retrieval system cover system architecture, data storage and representation, database and its remote access, and human-machine interfaces. The concept of physical and logical layers and events.

4. Media Types and Their Properties Audio Audio events can be defined as continuous audio segments that have some distinct acoustic characteristics Visual Text

5. Multimedia Data Abstraction and Retrieval A physical layer addresses the media events that are time continuous and better described using the specific characteristics of a media signal. A logical layer addresses multimedia events that present integrated behavior of other events across media.

6. Multimedia Data Abstraction and Retrieval(cont.) Physical Program and Physical Event Physical program a continuous stream of multimedia data that begins and ends due to the physical scene edits and cuts decided by human operators Physical event A continuous segment of multimedia data, representing a coherent and meaningful event within a physical program

7. Multimedia Data Abstraction and Retrieval(cont.) Logical Event A virtual view of the multimedia data constructed from a set of physical programs or events, or recursively from other physical or logical events. Application dependent Semantic Content Structure Conceptual and virtual Provide a seamless interface to user Table of Content (ToC)

8. Relationship among different event layers

9. System Architecture

10. System Implementation Server Implementation Content Hierarchy Extraction Layered Content Storage Store lower level information in a file system and higher level information in standard database systems When raw data is stored in a file system, different events in the data stream can be accessed using time based indexing Database Content Access Employ JDBC to provide connectivity to our database for query and update

11. Multimedia data storage in combined file system and database

12. Example of event indexing from database to a file

13. Database access

14. System Implementation(cont.) Client Browser Implementation Java applet-based interface Platform independence support for a variety of coded media Remote Database Query RMI (Remote Method Invocation) Browsing Interface Table of Content: a conceptual content structure TimeLine: a content structure in time Event Presentation: presentation and playback Content Playback with a JMF Player

15. Remote database access

16. Browser interface for a news program with both ToC and timeline representations of the semantic structure within the program

17. An Application: Multimedia Broadcast News Server End Content Hierarchy Extraction Broadcast news data is well structured. Lower levels – extract primitive physical events in each individual media stream. Higher semantic level – merge different events into semantically meaningful logical events. ex: Headline stories and news summaries.

18. Content hierarchy of broadcast news programs

19. Content Storage and Access In the file system, for each physical broadcast news program, store the timetables for audio, video,and text,individually. The tables in the database system will index into these timetables for the actual retrieval and data access. Physical events in each media are recorded in separate database tables at the physical event layer.

20. Relationship between logical event and physical event objects

21. Database design in broadcast news application

22. Client End Remote Database Query The physical program ID is uniquely determined by the channel and the date of the program. Content Presentation Content Structure Presentation ToC and TimeLine Static(single click) and playback(double click) Story Presentation News Summary Presentation Streaming Playback

23. Browser interface for a news program with both ToC and timeline representations of the semantic structure within the program

24. Visual representation for story “ El Nino ”

25. Representation for news summary of the day

26. Playback interface with synchronized audio/video/text

27. Summary We have introduced a new layered approach to multimedia content representation and storage for search and retrieval. The concepts are implemented in a client/server multimedia information retrieval system designed for broadcast news search and retrieval.