1. Video Trails: Representing and Visualizing Structure in Video Sequences Vikrant Kobla David Doermann Christos Faloutsos

2. Outline Background and Motivation Overview Video Trails Trail Segmentation Trail Classification Gradual Transition Detection Experiments and Results Conclusion

3. Background and Motivation Video is a valuable information resource There are still few efficient ways to provide access to the information the video contains Early work on indexing video treated video sequence as collections of still images, ignored the temporal structure Efficient analysis and representation of the temporal structure of a video is necessary

4. Overview 1. Generate a trail of points (Video Trails) in a low-dimensional space 2. Segment the video trails 3. Classify each of those segmented trails into two types: Stationary (low activity) VS Transitional (high activity) 4. Detect gradual transition

5. Video Trails Definition: A trail of points in a low-dimensional space where each point is derived from physical features of a single frame in the video clip Features: DC coefficients of the luminance and chrominance components of an MPEG frame Dimensionality Reduction (FastMap) initial feature vector a vector in that dimensional target dimension space FastMap

6. Example Consider a video clip with a 320x240 frame size Each frame has 20x15 MBs( Macroblock) Each MB contains 6 DC coefficients ( 4 luminance and 2 chrominance) Totally, 20x15x6=1800 coefficients (initial vector) 1800-by-1 vector (X1,X2,X3) 3 (target dimension) FastMap

7. Example

9. Trail Segmentation Segment the video in order to determine regions of high activity corresponding to transitions and low activity corresponding to individual shots The problem of segmenting the video into sets of frames is transformed into the problem of splitting the video trails into smaller trails corresponding to segments of video

10. Splitting Algorithm 1. Start by placing the first point in a new trail 2. Consider each successive point in the sequence in order 3. Perform a test for “inclusion” of this point in the current trail 4. if (the test pass) 5. Include the point in the current trail 6. Move to the next point 7. Goto 2 8. else 9. Close the current trail with the previous point as the last one 10. Start a new trail with only the current point 11. Goto 2

11. “Inclusion Test” Marginal Cost:Total cost per point in the trail Consider a clip with N frames Assume there are m points in the current trail, denoted by set, and be the point being considered for inclusion Define,d is the dimensionality So the new marginal cost is new marginal cost > previous one : not include new marginal cost < previous one : include

12. Example

13. Example (close-up)

14. The sequence of frames that yield the sparse transition between the two dense clusters

15. Trail Classification Classify each of those segmented trails into: Stationary (low activity) or Transitional (high activity) Classification Criteria – Monotonicity W1=0.4 – Sparsity W2=0.3 – Convex Hull Volume Ratio W3=0.2 – MBR Shape W4=0.1

16. Monotonicity If a trail is (close to) monotonic, in some direction,it’s likely transitional projection of distance along k projected distance ratio the length of MBR dimension k Minimum projected distance ratio

17. Monotonicity (Normalization) Recall: W1 is the weight of monotonicity Tlow is the lower bound=1.1 Tup is the upper bound=2.0

18. Sparsity Sparsity: total MBR volume per point Average Sparsity Sparsity Ratio Normalize

19. Convex Hull Volume Ratio The ratio of volume of the convex hull of points in a trail to the volume of MBR Normalize

20. MBR Shape Cuboidal Planar Elongated

21. Classification

22. Gradual Transition Detection Dissolves, Fades, Wipes Difficulty: activity arising from camera or large object motion also yields trails similar to trails resulting from gradual edits Filter out any kind of global motion leading to a transitional trail, Analysis global motion

23. Results

24. Conclusion Provide a compact representation of a video sequence structure Reduce a sequence MPEG frames to a trail of points in a low dimensional space Segment trails and classify each segment as either stationary or transitional Detect gradual edits