Video Table-of-Contents: Construction and Matching Master of Philosophy 3 rd Term Presentation - Presented by Ng Chung Wing

Outline Overview of Research Previous Work ADVISE Advanced Digital Video Information Segmentation Engine Future Work and Conclusion

Overview of Research Situation A large volume of video contents on the Internet Problems Not enough information to describe the video contents Difficult to search for videos with similar contents

Overview of Research Web-based Video Retrieval System Provides a Video Table-of-Contents to describe the structure of video Applies Tree Matching Algorithms to measure the similarity between videos Allows retrieval of similar videos according to tree matching results

Overview of Research (Cont’d) In the last semester Definition of Video Tree Structure Video Tree Matching Algorithms In this semester ADVISE  Generation of Video Tree Structure  Web-based presentation of the structure as a Video Table-of-Contents In the coming semester Video Retrieval

Review on Previous Work: Video Tree Structure Decompose a video into 5 levels: Video Frames Video Shots Video Groups Video Scenes Whole Video Hierarchical Representation of a Video

Review on Previous Work: Video Tree Structure (Cont’d) Example: Group 1 Group 3 Group 2 Scene 1 Scene 2 Video Shots:

Review on Previous Work: Video Tree Structure (Cont’d) 4 levels tree structure Regarded as: Video Table-of-Contents (V-ToC)

Review on Previous Work: Video Tree Matching Algorithms Measure the similarity of videos Matching on their video tree structures Two approaches: Ordered Tree Matching Algorithm  Constrained by temporal ordering Non-ordered Tree Matching Algorithm  Not constrained by temporal ordering Video feature used Color histograms of video frames

Review on Previous Work: Video Tree Matching Algorithms (Cont’d) Similarities in Algorithm Top-down manner (Scenes, Groups, Shots) Video features extraction at shot level Propagates result from bottom-up Differences in Algorithm Non-ordered Tree Matching Algorithm  Calculates the score of the best match at video, scene and group levels Ordered Tree Matching Algorithm  Calculates the score of the best ordered match using dynamic programming

ADVISE Advanced Digital Video Information Segmentation Engine 3 modules: Generates V-ToC to describe videos Presents V-ToC on the Internet using XML Allows video customization according to the V-ToC with SMIL

ADVISE: Video Structure Construction Video Shots Detection Color histogram based method with weighted regions  5 regional and 1 overall color histograms for each video frame  Catch local color features in video frame  Different weights to regions according to importance 6

ADVISE: Video Structure Construction (Cont’d)  Calculate the frame-to-frame color difference: where Hist i,t (k) denotes the k-th color value in the histogram for region i in frame t. W RDX are weights to regions.

ADVISE: Video Structure Construction (Cont’d)  Find the sudden change in color contents as the video shot boundaries  Need a threshold to determine the shot boundaries:  Not suitable to assign a fixed threshold to different videos Use adaptive threshold

ADVISE: Video Structure Construction (Cont’d)  Employed an entropic thresholding method Divide the frame-to-frame histograms difference into two populations at a threshold point Measure the entropies of the populations Find the maximum sum of two entropies at different threshold point  Distribution of histogram differences: 0Max. Difference Optimal Threshold (with most informative entropies) Shot Breaks Non-Shot Breaks

ADVISE: Video Structure Construction (Cont’d) Video Groups Formation For each shot s, compare its key frame (the first frame) with the key frame from most recent shot in group g.

ADVISE: Video Structure Construction (Cont’d) Shots closer in time are more similar  Consider the temporal factor:  where Temporal ts,tg is the temporal factor. AvgShotLength is the average length of video shots. K is a predefined value which controls the change in the temporal factor

ADVISE: Video Structure Construction (Cont’d) After comparing all groups, we assign the shot to a group if:  Difference is smallest amongst groups  Difference is smaller than the calculated threshold  The shot is temporally not far apart from the group

ADVISE: Video Structure Construction (Cont’d) Video Scenes Formation Construct a continuous video sequence from video groups Video scenes  For each group with the first and the last shots m and n  If m is within a scene, add the group to the scene and extend the scene to n if necessary. Case (i) & (ii)  If m is not within any scene, add it to a new scene Case (iii)

ADVISE: Video Structure Construction (Cont’d) User Interface of implemented system

ADVISE: Video Structure Construction (Cont’d) Experiments: Evaluate 4 different settings of video structure construction  Single color histogram, and fixed threshold  Single color histogram, and adaptive threshold  Weighted regional color histograms, and fixed threshold  Weighted regional color histograms, and adaptive threshold Compare the generated video structure with human judgments

ADVISE: Video Structure Construction (Cont’d) The setting with  Weighted regional color histograms  Adaptive threshold generates the most accurate video structure.

ADVISE: XML Presentation 4 benefits to store the video structure in XML Nested hierarchy  Fit into our video tree structure Plain-text format  Easy to search and modify Extensibility  Available to extend the video structure Application to the Internet

ADVISE: XML Presentation (Cont’d) Defined XML grammar for video structure in DTD DTD for XML Video Tree Structure XML Video Tree Structure

ADVISE: XML Presentation (Cont’d) Web-based presentation of XML using XSL Transformation to HTML Sorting and filtering of XML data at sec

ADVISE: XML Presentation (Cont’d) An example XML presentation V-ToC describes the structure of a video

ADVISE: SMIL Generation Video customization Allow user to pick some video segments that they are interested from the V-ToC SMIL Designed for performing synchronized multimedia presentation on the Internet Use RealPlayer to browse Benefits  Easy to generate because of the XML plain-text property  Dynamically adapt to different network and client condition

ADVISE: SMIL Generation (Cont’d) Defined a SMIL template:... Define the Layout <video src="rtsp:// source video on server" clip-begin="3s" clip-end="15s" region="video" fill="freeze"/> <textstream src="desc.rt" clip-begin="3s" clip-end="15s" region="description" fill="freeze"/> <video src="rtsp:// source video on server" clip-begin="35s" clip-end="50s" region="video" fill="freeze"/> <textstream src="desc.rt" clip-begin="35s" clip-end="50s" region="description" fill="freeze"/>

ADVISE: SMIL Generation (Cont’d) Customized SMIL Video Presentation Script on Web Server 1.Interpret request 2.Select video segments according to XML video structure 3.Generate customized SMIL presentation User Interface for Customization Submit request Return SMIL

Future Work Video retrieval system framework Integrates ADVISE and video tree matching algorithms Explore the capability of using the video tree matching on video retrieval Video clustering Efficient retrieval of video using XML Hierarchy of V-ToC Textual search of video information

Conclusion Overview of the research on video retrieval system Based on the structure of video (V-ToC) Described ADVISE Generates video tree structure (V-ToC) Provides V-ToC in XML as descriptions of videos on the Internet Enables video customization based on V-ToC using SMIL