1. ADVISE: Advanced Digital Video Information Segmentation Engine
Presented by Ng Chung Wing

2. Outline Introduction Overview of ADVISE Technologies in ADVISE
System architecture and services provided
Technologies in ADVISE
Construction of Video Table-of-Contents (V-ToC)
Video Summarization
Video Matching
Conclusion

3. Introduction
Videos is getting more popular in education, entertainment and information sharing
Evident growth of video contents on the Internet
57.2% of Internet users watched video chips and 7.3% edited video clips on their personal computers (Survey by PC Data, 2000)
Important task  retrieve an interested video!
Two problems in video retrieval:
Not enough information to describe the video contents
Difficult to search for videos with similar contents
Introduction

4. Introduction
We propose the “ADVISE” system to solve the above problems
ADVISE
Advanced Digital Video Information Segmentation Engine
Web-based video browsing and retrieval system
Provides a set of services:
For description of videos:
Video table-of-contents (VToC)
Video summarization
For searching similar videos:
Video feature similarity matching
Introduction

5. Contributions
We propose the framework of a video browsing and retrieval system called ADVISE
We build the image-based video description called Video Table-of-Contents (V-ToC) in ADVISE
We develop the Video Summarization Algorithm for generating video summaries in ADVISE
We propose two Video Tree Matching Algorithms, which measure the feature similarity between videos, in ADVISE
Introduction

6. Overview of ADVISE - Outline
Introduction
Overview of ADVISE
Objective of ADVISE
System Architecture
Services provided by ADVISE
Technologies in ADVISE
Conclusion

7. Objectives of ADVISE
To provide an efficient way to describe the video contents
To save the time for browsing the whole video to know the contents
To search videos with similarity in certain video features
To provide services through the Internet
Overview of ADVISE

8. System Architecture of ADVISE
ADVISE consists of 3 modules
(I) Video preprocessing module
(II) Web-based retrieval module
(III) Video streaming server
 Major contributions
 Process source videos
 Provide services to users of ADVISE
 Setup the Real System Server for delivering video contents
(III) Video streaming server
 Setup the Real System Server for delivering video contents
Overview of ADVISE

9. System Architecture of ADVISEII I
III
Overview of ADVISE

10. (II) Web-based Video Retrieval Module
User interface for accessing services of ADVISE
This module reside on a web server
There are 3 services provided by ADVISE 1
Service 1: V-ToC Presentation
Service 2: Generation of SMIL Video Summary
Service 3: Querying Similar Videos 2 3
Overview of ADVISE

11. Module (II) ~ Service 1: V-ToC Presentation
Image-based description for the video content
Use the V-ToC structure resulting from Module (I)
Used XML with XSL to generate a flexible web-based presentation
Each V-ToC show us the contents organization in a video
Overview of ADVISE

12. Module (II) ~ Service 2: Generation of SMIL Video Summary
Generate a video summary according to user’s preference
Used SMIL to deliver the customized video summary to the user
User’s input
Resulting SMIL Video Summary
Overview of ADVISE

13. Module (II) ~ Service 3: Querying Similar Videos
Show similar videos in descending order of the similarity score
Results of video matching in Module (I)
User can select matching different video features
Color histogram feature
Shot style feature
List of videos with different
similarity scores with the query
video
Overview of ADVISE

14. Technologies in ADVISE - Outline
Introduction
Overview of ADVISE
Technologies in ADVISE
For service 1: Construction of Video Table-of-Contents (V-ToC)
For service 2: Video Summarization
For service 3: Video Matching
Conclusion

15. Construction of Video Table-of-Contents (V-ToC)
Uses
Image-based video description which show the organization of video contents
 Video Table-of-Contents (V-ToC)
Provide the hierarchy for structural matching of video
Video structure used in ADVISE
Hierarchical tree structure with 4 levels
Storage and Presentation
Use XML and XSL
Technologies in ADVISE - Construction of V-ToC

16. Video Structure in ADVISE
Decompose a video into 5 levels:
Video Frames
Video Shots
Video Groups
Video Scenes
Whole Video
Hierarchical Representation of a Video
Technologies in ADVISE - Construction of V-ToC

17. Video Structure in ADVISE
Example: 2
Shots: 1
Scene 1
Video 3
Group 1 4
Shot 1
Scene 2
Group 2
Shots 2,4,6
Group 3
Shots 3,5,7
Technologies in ADVISE - Construction of V-ToC

18. Video Structure in ADVISE
Structure videos from the bottom level
5 steps in video structuring
i. Color Histograms Extraction
ii. Video Shot Boundaries Detection
iii. Video Groups Formation
iv. Video Scenes Formation
Technologies in ADVISE - Construction of V-ToC

19. Storage and Presentation
Resulting Presentation of V-ToC using XML and XSL
<?xml version="1.0"?>
<!DOCTYPE advise SYSTEM "./toc.dtd">
<advise>
<video length ="25"
src="rstp://localhost/video1.rm">
<scene id="1">
<group id="1">
<shot id="1">
<keyframe img="./sh_1.jpg"/>
<time value="0"/>
</shot>
<shot id="2">
<keyframe img="./sh_2.jpg"/>
<time value="11"/>
</group>
</scene>
</video>
</advise>
Technologies in ADVISE - Construction of V-ToC

20. Technologies in ADVISE - Outline
Introduction
Overview of ADVISE
Technologies in ADVISE
For service 1: Construction of Video Table-of-Contents (V-ToC)
For service 2: Video Summarization
For service 3: Video Matching
Conclusion

21. Video Summarization
User may still not be able to know the exact video contents with V-ToC
Video summary can provide all types of information in the video
Objectives:
Select the major contents
Shorten the duration for browsing
Difficulties
No standard method to pick the important contents from video
Importance of contents depends on user’s need
In ADVISE:
We accept user’s input for generating video summary such that the result can be the best suitable for the user
Technologies in ADVISE Video Summarization

22. Inputs for Video Summarization Algorithm
Video features used:
Human faces
Male and female voices
Volume level
Caption text
User’s inputs for customization of the video summary
Weights of different video features
Time constraint for video summary
Clustering control constant
Technologies in ADVISE Video Summarization

23. Video Summarization Algorithm
4 steps to summarize a video
i. Combining extracted video segments
ii. Scoring the extracted video segments
iii. Selecting extracted video segments
iv. Refining the selection result
Example:
Step (i)
Step (ii)
Step (iii)
Step (iv)
Technologies in ADVISE Video Summarization

24. Video Summary in SMIL
SMIL presentation are delivered to user of ADVISE
Can be generated instantly
Can be browsed by the user on the Internet using a stream-based protocol
Resulting SMIL video summary
Technologies in ADVISE Video Summarization

25. Technologies in ADVISE - Outline
Introduction
Overview of ADVISE
Technologies in ADVISE
For service 1: Construction of Video Table-of-Contents (V-ToC)
For service 2: Video Summarization
For service 3: Video Matching
Conclusion

26. Video Matching Video Matching VToC is a tree structure
Match the extracted video features
Color, motion, shape, etc.
Sequential matching
Non related to video structure
VToC is a tree structure
Can apply tree matching algorithm
Matching related to video structure
In ADVISE, we propose two tree matching algorithms
(1) Non-ordered tree matching algorithm
(2) Ordered tree matching algorithm (Consider temporal ordering)
Technologies in ADVISE Video Matching

27. Input Features for Video Matching
Two video features used
Color histograms feature
Take the first frame of a video shot as the key frame to compare in order to reduce the computational complexity.
Compare the visual similarity.
Shot style feature
Compose of camera motion and length of a video shot.
Select the first camera motion in a video shot as the representative.
Compare the similarity in video pace.
Technologies in ADVISE Video Matching

28. (1) Non-ordered Tree Matching Algorithm
Not constrained by temporal ordering
Capture all similar components
Algorithm
Extract features at the bottom level
Propagate the similarity score up to the root level
Group a1
Group b2
Video A
Video B
Group a2
Group b1
Technologies in ADVISE Video Matching

29. (2) Ordered Tree Matching Algorithm
Constrained by temporal ordering
Temporal ordering can affect the video contents
Reduce the problem to match components in order
Capture only ordered similar components
Group a1
Group b2
Video A
Video B
Group a2
Group b1
Technologies in ADVISE Video Matching

30. (2) Ordered Tree Matching Algorithm
Recursive dynamic programming
Hierarchical matching
From the tree root (video level)
Until shot level, extract the video feature similarity
Reduced the complexity compare with approach (1)
Technologies in ADVISE Video Matching

31. Conclusion
The ADVISE system, which enhanced video browsing and retrieval system on the Internet, is proposed.
The generation and presentation of the image-based video description are developed.
The automation of video summarization into SMIL format is provided.
Two video tree matching algorithms for measuring the similarity between videos are proposed.
Conclusion

32. Questions
&
Answers