Introduction To Computational and Biological Vision Max Binshtok Ohad Greenshpan March 2006 Shot Detection in video

What is a shot ? Shot: A drastic change in successive frames in a video.

Is “shot detection” easy ? Know the enemy: –Change in luminance –Rapid motion (object, camera) –Blur (influence on edges) –Graphical special effects Is the term “Shot” well defined ? Fade-in, fade-out Large objects blocking the scene

Motivation Video Indexing Guarding Cameras (Alerts) Catching “Priming” in commercials

Our Tool Learning Experience Frames Sequence Image Shot Detection (all successive images) One-To-One comparison Single Image processing

FlowPixel-to-PixelHistogramHausdorff Supervised Learning Input

Pixel To Pixel (P2P) Method General Idea: Summing the differences between the pixels’ intensity values over the whole image (reminds the similarity measure – SSD). Variants: –Global P2P –Cumulative P2P

Global P2P Formulation : “Shot Detection” if Disadvantage : ignores local changes Cumulative P2P Formulation : “Shot Detection” if Disadvantage: sensitivity to motion Time complexity of both: O( |pixels| )

Global P2P Results

1: 2: 3: good …

Cumulative P2P Results

1: 2: 3: 4:4:4:4: 5:5:5:5:

Histogram-Based Methods General Idea: –Sums the differences between the images’ histograms Variants: –Simple histogram –Max histogram –Weighted histogram

Simple Histogram Method Formulation: “Shot Detection” if Problems with this method: –Sensitivity to “lighting” and colors change Time Complexity: O( |pixels| + |bins| )

Simple Histogram Results

1: 2: 3: 4:4:4:4: Weird !! Worth Checking … 1.Does not look like a shot 2.Was not detected by P2P

Let’s Compare These Two Images A close examination shows that the sun showed up at frame 56 and lightened the field for the following images… Due to the fact that the clay color is very dominant, the sun changed the images histograms.

Max Histogram Method Formulation: “Shot Detection” if Time Complexity: O( |pixels| + |bins| ) Problems with this method: –Same sensitivity to lighting and color change, but takes into account all colors (R,G,B) –Thus, might over-influenced by a change in one color …

Simple Histogram Vs. Max Histogram Same results except one..

Weighted Histogram Method Formulation: “Shot Detection” if Problems with this method: –Same as the formers Time Complexity: O( |pixels| + |bins| )

Hausdorff Method General Idea: –Mark Edge points (using an edge detector) –Calculate the proportion of outlying edge points – “Outliers”. Notes: –“Outlier” : An edge points which does not have a corresponding edge point in a pre- defined neighborhood, in the other image. –Edge detector we have implemented: Sobel. (embedded smoothing effect)

Hausdorff Method Disadvantages: –Edge points might correlate with other edge points. –Sensitivity to the edge detector threshold (i.e.: fading, noise, special effects) Time Complexity: O( |pixels| ) Dominated by the edge detection process.

Hausdorff Method Two interesting “false-detections”:

Hausdorff Method Let’s check their Sobel results: Sobel results: Sensitivity to the edge detector’s threshold..

LearningPixel-to-PixelHistogramHausdorff Neural Network Yes / No

Learning Main Advantages: –Combination of methods –Threshold values are not important –Ability to adapt to new types of data sets Results: –Learning is improving –Learning is general

Time for “Hands On” ?

Questions ?