ADVISE: Advanced Digital Video Information Segmentation Engine Presented by Ng Chung Wing

Outline Introduction Overview of ADVISE Technologies in ADVISE System architecture and services provided Technologies in ADVISE Construction of Video Table-of-Contents (V-ToC) Video Summarization Video Matching Conclusion

Introduction Videos is getting more popular in education, entertainment and information sharing Evident growth of video contents on the Internet 57.2% of Internet users watched video chips and 7.3% edited video clips on their personal computers (Survey by PC Data, 2000) Important task  retrieve an interested video! Two problems in video retrieval: Not enough information to describe the video contents Difficult to search for videos with similar contents Introduction

Introduction We propose the “ADVISE” system to solve the above problems ADVISE Advanced Digital Video Information Segmentation Engine Web-based video browsing and retrieval system Provides a set of services: For description of videos: Video table-of-contents (VToC) Video summarization For searching similar videos: Video feature similarity matching Introduction

Contributions We propose the framework of a video browsing and retrieval system called ADVISE We build the image-based video description called Video Table-of-Contents (V-ToC) in ADVISE We develop the Video Summarization Algorithm for generating video summaries in ADVISE We propose two Video Tree Matching Algorithms, which measure the feature similarity between videos, in ADVISE Introduction

Overview of ADVISE - Outline Introduction Overview of ADVISE Objective of ADVISE System Architecture Services provided by ADVISE Technologies in ADVISE Conclusion

Objectives of ADVISE To provide an efficient way to describe the video contents To save the time for browsing the whole video to know the contents To search videos with similarity in certain video features To provide services through the Internet Overview of ADVISE

System Architecture of ADVISE ADVISE consists of 3 modules (I) Video preprocessing module (II) Web-based retrieval module (III) Video streaming server  Major contributions  Process source videos  Provide services to users of ADVISE  Setup the Real System Server for delivering video contents (III) Video streaming server  Setup the Real System Server for delivering video contents Overview of ADVISE

System Architecture of ADVISE II I III Overview of ADVISE

(II) Web-based Video Retrieval Module User interface for accessing services of ADVISE This module reside on a web server There are 3 services provided by ADVISE 1 Service 1: V-ToC Presentation Service 2: Generation of SMIL Video Summary Service 3: Querying Similar Videos 2 3 Overview of ADVISE

Module (II) ~ Service 1: V-ToC Presentation Image-based description for the video content Use the V-ToC structure resulting from Module (I) Used XML with XSL to generate a flexible web-based presentation Each V-ToC show us the contents organization in a video Overview of ADVISE

Module (II) ~ Service 2: Generation of SMIL Video Summary Generate a video summary according to user’s preference Used SMIL to deliver the customized video summary to the user User’s input Resulting SMIL Video Summary Overview of ADVISE

Module (II) ~ Service 3: Querying Similar Videos Show similar videos in descending order of the similarity score Results of video matching in Module (I) User can select matching different video features Color histogram feature Shot style feature List of videos with different similarity scores with the query video Overview of ADVISE

Technologies in ADVISE - Outline Introduction Overview of ADVISE Technologies in ADVISE For service 1: Construction of Video Table-of-Contents (V-ToC) For service 2: Video Summarization For service 3: Video Matching Conclusion

Construction of Video Table-of-Contents (V-ToC) Uses Image-based video description which show the organization of video contents  Video Table-of-Contents (V-ToC) Provide the hierarchy for structural matching of video Video structure used in ADVISE Hierarchical tree structure with 4 levels Storage and Presentation Use XML and XSL Technologies in ADVISE - Construction of V-ToC

Video Structure in ADVISE Decompose a video into 5 levels: Video Frames Video Shots Video Groups Video Scenes Whole Video Hierarchical Representation of a Video Technologies in ADVISE - Construction of V-ToC

Video Structure in ADVISE Example: 2 Shots: 1 Scene 1 Video 3 Group 1 4 Shot 1 Scene 2 Group 2 Shots 2,4,6 Group 3 Shots 3,5,7 Technologies in ADVISE - Construction of V-ToC

Video Structure in ADVISE Structure videos from the bottom level 5 steps in video structuring i. Color Histograms Extraction ii. Video Shot Boundaries Detection iii. Video Groups Formation iv. Video Scenes Formation Technologies in ADVISE - Construction of V-ToC

Storage and Presentation Resulting Presentation of V-ToC using XML and XSL <?xml version="1.0"?> <!DOCTYPE advise SYSTEM "./toc.dtd"> <advise> <video length ="25" src="rstp://localhost/video1.rm"> <scene id="1"> <group id="1"> <shot id="1"> <keyframe img="./sh_1.jpg"/> <time value="0"/> </shot> <shot id="2"> <keyframe img="./sh_2.jpg"/> <time value="11"/> </group> </scene> </video> </advise> Technologies in ADVISE - Construction of V-ToC

Technologies in ADVISE - Outline Introduction Overview of ADVISE Technologies in ADVISE For service 1: Construction of Video Table-of-Contents (V-ToC) For service 2: Video Summarization For service 3: Video Matching Conclusion

Video Summarization User may still not be able to know the exact video contents with V-ToC Video summary can provide all types of information in the video Objectives: Select the major contents Shorten the duration for browsing Difficulties No standard method to pick the important contents from video Importance of contents depends on user’s need In ADVISE: We accept user’s input for generating video summary such that the result can be the best suitable for the user Technologies in ADVISE - Video Summarization

Inputs for Video Summarization Algorithm Video features used: Human faces Male and female voices Volume level Caption text User’s inputs for customization of the video summary Weights of different video features Time constraint for video summary Clustering control constant Technologies in ADVISE - Video Summarization

Video Summarization Algorithm 4 steps to summarize a video i. Combining extracted video segments ii. Scoring the extracted video segments iii. Selecting extracted video segments iv. Refining the selection result Example: Step (i) Step (ii) Step (iii) Step (iv) Technologies in ADVISE - Video Summarization

Video Summary in SMIL SMIL presentation are delivered to user of ADVISE Can be generated instantly Can be browsed by the user on the Internet using a stream-based protocol Resulting SMIL video summary Technologies in ADVISE - Video Summarization

Technologies in ADVISE - Outline Introduction Overview of ADVISE Technologies in ADVISE For service 1: Construction of Video Table-of-Contents (V-ToC) For service 2: Video Summarization For service 3: Video Matching Conclusion

Video Matching Video Matching VToC is a tree structure Match the extracted video features Color, motion, shape, etc. Sequential matching Non related to video structure VToC is a tree structure Can apply tree matching algorithm Matching related to video structure In ADVISE, we propose two tree matching algorithms (1) Non-ordered tree matching algorithm (2) Ordered tree matching algorithm (Consider temporal ordering) Technologies in ADVISE - Video Matching

Input Features for Video Matching Two video features used Color histograms feature Take the first frame of a video shot as the key frame to compare in order to reduce the computational complexity. Compare the visual similarity. Shot style feature Compose of camera motion and length of a video shot. Select the first camera motion in a video shot as the representative. Compare the similarity in video pace. Technologies in ADVISE - Video Matching

(1) Non-ordered Tree Matching Algorithm Not constrained by temporal ordering Capture all similar components Algorithm Extract features at the bottom level Propagate the similarity score up to the root level Group a1 Group b2 Video A Video B Group a2 Group b1 Technologies in ADVISE - Video Matching

(2) Ordered Tree Matching Algorithm Constrained by temporal ordering Temporal ordering can affect the video contents Reduce the problem to match components in order Capture only ordered similar components Group a1 Group b2 Video A Video B Group a2 Group b1 Technologies in ADVISE - Video Matching

(2) Ordered Tree Matching Algorithm Recursive dynamic programming Hierarchical matching From the tree root (video level) Until shot level, extract the video feature similarity Reduced the complexity compare with approach (1) Technologies in ADVISE - Video Matching

Conclusion The ADVISE system, which enhanced video browsing and retrieval system on the Internet, is proposed. The generation and presentation of the image-based video description are developed. The automation of video summarization into SMIL format is provided. Two video tree matching algorithms for measuring the similarity between videos are proposed. Conclusion

Questions & Answers