2. Presenter: Ibrahim A. Zedan
Abrupt Cut Detection in News Videos using Dominant Colors Representation
Presenter: Ibrahim A. Zedan

3. Agenda Introduction SBD Foundation, Challenges, and Methods
News Video Structure
Shot Boundary Detection (SBD) Definition
Video Shot vs. Video Scene
Shot Transition Types
SBD Foundation, Challenges, and Methods
Proposed System
Dominant Colors Representation
Dissimilarity Feature Vector Calculation
Feed Forward Neural Network Simulation
Experimental Results
Conclusion and Future Work

4. Introduction

5. Video Indexing
The amazing increase in video data on the internet leads to an urgent need of summarization, indexing, and retrieval of this video data.

6. News Video News videos have gained a special importance.
Many individuals are concerned about news video.
Gives us the main news in the world recently.
A set of semantically independent stories.
News story
An arrangement of shots with a coherent focus which contain no less than two independent declarative clauses.

7. News Video Structure

8. Shot Boundary Detection (SBD) Temporal Video Segmentation
Video shot
A series of consecutive interrelated frames represents continuous activities in time and space and recorded contiguously by a single camera.
Shot Boundary Detection
A method automatically identifying the shots transitions inside a video.

9. Video Shot vs. Video Scene
A logical group of temporally adjacent shots with a common location, a common object of interest or a common thematic concept.

10. Shot Transition Types

11. Abrupt Cut
Instantaneous transition from one shot to the next shot that occurs in a single frame.

12. Fade-In
A progressive transition in which shot begins with a single color frame and develops slowly till it illuminates to its full force.

13. Fade-Out
The converse of a fade-in.

14. Dissolve
The interfering of two shots where both shots are slightly apparent.
During the transition the previous shot share gradually decreases while the next shot share increases.

15. Dissolve Cont.

16. Wipe
A pattern moves across the screen, progressively replacing the previous shot with the next shot.

17. Shot Boundary Detection Foundation, Challenges, and Methods

18. SBD Foundation Detecting the Frames Discontinuities
The SBD Performance relies on:
Choice of the Similarity Measure
Threshold Selection

19. SBD Challenges SBD Challenges: Gradual Transitions
Illumination changes
Flash Lights
Object/Camera Motion
Smoke, fire
Video in video (Screen split)

20. Shot Boundary Detection Methods

21. Shot Boundary Detection Methods Cont.
Pixel-Based
Easy to implement.
Very sensitive to motion and noise.
Statistical-Based (Ex: Histogram)
Spatial information is missed.
Calculated on the entire frame level or at a block level.
Edge Based
Entering and exiting edges.
Execution time !
Motion-Based
Not favored in the uncompressed domain.
Huge computational power.

22. Proposed System

23. Main Idea
A shot boundary exist  large difference of the dominate colors order.
Simulating the neural network with pairs of consecutive frames.

24. Dominant Colors Representation
Elimination of the candidate caption area.
Captions in news videos disrupt both the SBD process and key frames extraction.
Image is represented by the order of its colors sorted from the highest frequency to the lowest frequency.
Colors Quantization
Absorb the lighting conditions that result in variation of the pixels color intensities of the same object.

25. Dominate Color Representation (RGB)

26. Dominate Color Representation (Gray)

27. Dissimilarity Feature Vector
The Dissimilarity between two images is a vector contains the difference in order for each color.
Similar Frames  Similar order of dominate colors.
Different frames  Large difference of the dominate colors order.

28. Dissimilarity Feature Vector Calculation

29. Feed Forward Neural Network Training
The neural network is trained by the feature vectors of the consecutive frames dissimilarity.
Building and training of a neural network to differentiate between :
Similar Frames  Consecutive frames belong to the same video shot.
Different Frames  Consecutive frames belong to abrupt cuts.
L1 = sqrt((M+2)*N)+2*sqrt(N/(M+2)) (1)
L2 = M*sqrt(N/(M+2)) (2)

30. Abrupt Cut Detection Process

32. Experimental Results

33. Manual Shot Boundary Marker Tool

34. Data Set Statistics 1040 # of Abrupt Cuts 53 # of Videos 268
# of Gradual Transitions 85
Total Duration in minutes

35. Evaluation Criteria
We carried out several experiments with the dominate colors gray representation and RGB representation with several quantization steps.
Precision (P) = # Correctly Detected Cuts / # Detected Cuts
Recall (R) = # Correctly Detected Cuts / # Ground Truth Cuts
F Measure = 2 * Precision * Recall / ( Precision + Recall )

36. Gray vs. RGB Representation
Quantization
Step
Gray Representation
RGB Representation
F Measure 2
91.00 %
91.56 % 4
91.08 %
93.40 % 8
92.50 %
93.30 % 16
91.36 %
91.40 % 32
86.83 %
89.49 % 64
63.00 %
82.05 %

37. Detailed Results without Flash Light Elimination
Video
Ground Truth
# Cuts
Correct
Miss
False Alarms P % R F
Vid1 68 63 5 9
87.50
92.65
90.00
Vid2 27 26 1 2
92.86
96.30
94.55
Vid22 23
100.00
100.0
Vid30 25
83.33
96.15
89.29
Vid43 4
86.21
92.60
Vid9 29
Total
198
191 7 20
90.52
96.46
93.40

38. False Alarms Reasons
Flash Light
Rapid Motion

39. False Alarms Reasons Cont.
Dissolves especially short dissolves.

40. Misses Observation
Same Scene (Similar background)

41. Flash Lights Elimination with Flash Light Elimination
Video
Ground Truth
# Cuts
Correct
Miss
False Alarms P % R F
Vid1 68 62 6 8
88.57
91.17
89.86
Vid2 27 26 1 2
92.86
96.30
94.55
Vid22 23
100.00
100.0
Vid30 25
92.59
96.15
94.34
Vid43
Vid9 29
Total
198
190 12
94.06
95.96
95.00

42. Conclusion and Future Work

43. Conclusion
The new image representation and dissimilarity measure succeeded to detect the abrupt cuts in news videos with promising accuracy.
The training data is not enough.
The proposed system is tested on challenging data.
As quantization step increase the accuracy increased up to limit.
The RGB dominant colors representations have slightly better results than the gray scale representation.
Flash light effect introduced in two successive frames at most.

44. Future Work Extend the data set.
Retrain the models to enhance results.
Add gradual transition detector.

45. Cite this paper as
Zedan, I.A., Elsayed, K.M., Emary, E.: Abrupt cut detection in news videos using dominant colors representation. In: Proceedings of the 2nd International Conference on Advanced Intelligent Systems and Informatics (AISI 2016), Advances in Intelligent Systems and Computing, vol. 533, pp. 320–331, Cairo, Egypt, 24–26 Oct 2016

46. Questions ?