Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Introduction Recognition and Segmentation Min Cut Max Flow Single Image Methods –GrabCut –Lazy Snapping –…

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Lazy Snapping Interactive User Interface

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Lazy Snapping Energy minimization

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Lazy Snapping Energy minimization

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Lazy Snapping Boundary overriding

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Lazy Snapping Boundary overriding

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Motivation Obvious Next Step Video Cut & Paste Video Manipulation and Editing

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Introduction Frame by Frame –Time Consuming and Tedious Error With Simple Methods –Fast motions –Deforming silhouettes –Changing topologies

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Introduction Two Papers –Video Object Cut and Paste –Video Cutout

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Object Cut and Paste Yin Li, Jian Sun, Heung-Yeung Shum

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Overview

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Pre-segmentation Pre-Segmentation to All Frames

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Key Frames Picking Key Frames.

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Key Frames User Fore/Background Segmentation

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Graph Cut Segmentation 3D Graph – G=(V,A) Labeling –Foreground = 1 –Background = 0 Volume Between Successive Key Frames

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Graph Construction 2 Kinds of Arcs: –A I – Intra Frames (BLUE) –A T – Inter Frame (RED)

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Graph Construction Minimizing Equation: E 1 – Global Color Models E 2 – Penalizing Spatially E 3 – Penalizing Temporally

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Likelihood Energy GMMs Decide Label In Key Frames:

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon GMM Gaussian Mixture Model Distance is Measured By:

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Prior Energies E 2, E 3 Are the Same Distance of Adjacent Regions. β = (2 E (||c r – c s || 2 )) -1

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Prior Energies λ 1 = 24 λ 2 = 12

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Graph Segmentation

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Errors Global Colors Similarity to Background Thin Areas

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Error Overriding Video tubes Manual corrections

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Tubes Local Color Models Put Two Windows Tracking Algorithm Key Frames to Solve W1W1 WTWT

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Fixing graph cut segmentation Minimizing:

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Overriding Brush Fixing Boundary Manually

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Manual error overriding

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Soften hard segmentation Matting

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Coherent Matting Boundary is not 0/1 Prevent Bolting Pixels Smooth Paste

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Coherent Matting

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Example

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Example

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Example

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Example

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video CutOut J. W ANG, P. B HAT, A. C OLBURN, M. A GRAWALA, M. C OHEN. Interactive Video Cutout. ACM Trans. on Graphics (Proc. of SIGGAPH2005), 2005

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Cutout introduction What’s new? Different user interface 3D graph formation Refinement mechanism

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon System overview

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Graph construction Hierarchical graph nodes: 1.Frame by frame mean shift segmentation 2.Aggregating segments across frames

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Pixel 26-neighborhood induce links Lower level links induce higher level link 3D Graph construction

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Stroking foreground and background over the 3D spatio-temporal volume Not segmenting any frame User Interface

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Graph construction –User input propagates upward –Min cut uses yellow nodes 3D Min cut/Max flow

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Min cut/Max flow Weights / Energy function –The energy function: –Data term: color similarity to F/B model –Link term: cut likelihood

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Min cut/Max flow Terms in energy function Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G ColorL L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Data weight User input generates color model (GMM) Infinite weight preserves marked pixels Data weight = abiding to F/B color model Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. history D B,G ColorD F,G Color L L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Data weight White – high probability Foreground Black – Low probability Foreground Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G Color D F,G Color L L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Data weight White – high probability Background Black – Low probability Background Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. history D B,G Color D F,G ColorL L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Strong gradients segment border Link cost encourage cut at edges Link weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G ColorL L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Link weight White – low cut probability Black – high cut probability Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G ColorL L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Pixel span: (x o, y o, t) t>0 Data weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. history D B,G ColorD F,G ColorL L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Data weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. history D B,G ColorD F,G ColorL L Local temporal L G Gradient Local background model Assuming camera is stabilized, video is registered Extracting “clean plate” Weight per pixel span

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon d(z i ) = minimum color distance {“clean plate”, B marked pixel}. “Clean plate” cannot be always trusted Weight: Data weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. history D B,G ColorD F,G ColorL L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Data weight White – high probability Background Black – Low probability Background Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. history D B,G ColorD F,G ColorL L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Link span: links between two adjacent pixel spans Link weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G Color L L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Strong edges exists within segment Small change over time Local temporal link cost penalize strong temporal gradient Link weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G Color L L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Link weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G Color L L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Link weight White – low cut probability Black – high cut probability Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G Color L L Local temporal L G Gradient

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Energy function Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G ColorL L Local temporal L G Gradient 3D Min cut/Max flow λ2λ2 λ1λ1 λ3λ3

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Iterative process The user refines the cut Adds F/B strokes Graph is re-computed N+1 th iterationN th iteration

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Post Processing

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Post processing Binary cut obtained Edges need refinement

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon A pixel-level min cut around edges Color model obtained form boundary Uniform edge cost = small cut Refinement

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Matting Soften hard segmentation Evaluate α Channel

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Matting Refinement fixed boundary locally Global 3D mesh α Channel along mesh normals

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Results

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Results

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Performance

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Pros –Online 3D min cut –Spatio temporal smooth cut Cons –Does not handle shadows –Ignore motion blur (LPF to avoid temporal aliasing) –Cannot separate translucent objects Summary

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Comparison Video Cutout Video object cut and paste Features 3D segmentation in 2 stages spatial-temporal manipulation 2D segmentation Frame base interface Graph nodes UI Performance 25 min 10 sec per Min cut 30 min 60 min 4-5 min 25 min ? 30 min Preprocessing Artist time Post processing Total

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Questions?

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon The total energy: Foreground and background terms: Background terms: Link terms: Energy function