Presentation is loading. Please wait.

Presentation is loading. Please wait.

Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006.

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006."— Presentation transcript:

1 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006

2 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Introduction Recognition and Segmentation Min Cut Max Flow Single Image Methods –GrabCut –Lazy Snapping –…

3 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Lazy Snapping Interactive User Interface

4 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Lazy Snapping Energy minimization

5 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Lazy Snapping Energy minimization

6 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Lazy Snapping Boundary overriding

7 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Lazy Snapping Boundary overriding

8 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Motivation Obvious Next Step Video Cut & Paste Video Manipulation and Editing

9 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Introduction Frame by Frame –Time Consuming and Tedious Error With Simple Methods –Fast motions –Deforming silhouettes –Changing topologies

10 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Introduction Two Papers –Video Object Cut and Paste –Video Cutout

11 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Object Cut and Paste Yin Li, Jian Sun, Heung-Yeung Shum

12 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Overview

13 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Pre-segmentation Pre-Segmentation to All Frames

14 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Key Frames Picking Key Frames.

15 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Key Frames User Fore/Background Segmentation

16 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Graph Cut Segmentation 3D Graph – G=(V,A) Labeling –Foreground = 1 –Background = 0 Volume Between Successive Key Frames

17 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Graph Construction 2 Kinds of Arcs: –A I – Intra Frames (BLUE) –A T – Inter Frame (RED)

18 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Graph Construction Minimizing Equation: E 1 – Global Color Models E 2 – Penalizing Spatially E 3 – Penalizing Temporally

19 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Likelihood Energy GMMs Decide Label In Key Frames:

20 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon GMM Gaussian Mixture Model Distance is Measured By:

21 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Prior Energies E 2, E 3 Are the Same Distance of Adjacent Regions. β = (2 E (||c r – c s || 2 )) -1

22 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Prior Energies λ 1 = 24 λ 2 = 12

23 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Graph Segmentation

24 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Errors Global Colors Similarity to Background Thin Areas

25 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Error Overriding Video tubes Manual corrections

26 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Tubes Local Color Models Put Two Windows Tracking Algorithm Key Frames to Solve W1W1 WTWT

27 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Fixing graph cut segmentation Minimizing:

28 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Overriding Brush Fixing Boundary Manually

29 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Manual error overriding

30 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Soften hard segmentation Matting

31 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Coherent Matting Boundary is not 0/1 Prevent Bolting Pixels Smooth Paste

32 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Coherent Matting

33 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Example

34 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Example

35 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Example

36 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Example

37 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video CutOut J. W ANG, P. B HAT, A. C OLBURN, M. A GRAWALA, M. C OHEN. Interactive Video Cutout. ACM Trans. on Graphics (Proc. of SIGGAPH2005), 2005

38 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Cutout introduction What’s new? Different user interface 3D graph formation Refinement mechanism

39 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon System overview

40 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Graph construction Hierarchical graph nodes: 1.Frame by frame mean shift segmentation 2.Aggregating segments across frames

41 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Pixel 26-neighborhood induce links Lower level links induce higher level link 3D Graph construction

42 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Stroking foreground and background over the 3D spatio-temporal volume Not segmenting any frame User Interface

43 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Graph construction –User input propagates upward –Min cut uses yellow nodes 3D Min cut/Max flow

44 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Min cut/Max flow Weights / Energy function –The energy function: –Data term: color similarity to F/B model –Link term: cut likelihood

45 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon 3D Min cut/Max flow Terms in energy function Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G ColorL L Local temporal L G Gradient

46 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Data weight User input generates color model (GMM) Infinite weight preserves marked pixels Data weight = abiding to F/B color model Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. history D B,G ColorD F,G Color L L Local temporal L G Gradient

47 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Data weight White – high probability Foreground Black – Low probability Foreground Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G Color D F,G Color L L Local temporal L G Gradient

48 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Data weight White – high probability Background Black – Low probability Background Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. history D B,G Color D F,G ColorL L Local temporal L G Gradient

49 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Strong gradients segment border Link cost encourage cut at edges Link weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G ColorL L Local temporal L G Gradient

50 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Link weight White – low cut probability Black – high cut probability Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G ColorL L Local temporal L G Gradient

51 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Pixel span: (x o, y o, t) t>0 Data weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. history D B,G ColorD F,G ColorL L Local temporal L G Gradient

52 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Data weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. history D B,G ColorD F,G ColorL L Local temporal L G Gradient Local background model Assuming camera is stabilized, video is registered Extracting “clean plate” Weight per pixel span

53 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon d(z i ) = minimum color distance {“clean plate”, B marked pixel}. “Clean plate” cannot be always trusted Weight: Data weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. history D B,G ColorD F,G ColorL L Local temporal L G Gradient

54 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Data weight White – high probability Background Black – Low probability Background Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. history D B,G ColorD F,G ColorL L Local temporal L G Gradient

55 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Link span: links between two adjacent pixel spans Link weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G Color L L Local temporal L G Gradient

56 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Strong edges exists within segment Small change over time Local temporal link cost penalize strong temporal gradient Link weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G Color L L Local temporal L G Gradient

57 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Link weight Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G Color L L Local temporal L G Gradient

58 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Link weight White – low cut probability Black – high cut probability Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G Color L L Local temporal L G Gradient

59 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Energy function Graph Cut Energy function L Link D F Foreground D B Background D B,L Pix. historyD B,G ColorD F,G ColorL L Local temporal L G Gradient 3D Min cut/Max flow λ2λ2 λ1λ1 λ3λ3

60 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Iterative process The user refines the cut Adds F/B strokes Graph is re-computed N+1 th iterationN th iteration

61 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Post Processing

62 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Post processing Binary cut obtained Edges need refinement

63 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon A pixel-level min cut around edges Color model obtained form boundary Uniform edge cost = small cut Refinement

64 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon

65 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Matting Soften hard segmentation Evaluate α Channel

66 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Matting Refinement fixed boundary locally Global 3D mesh α Channel along mesh normals

67 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Results

68 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Results

69 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Performance

70 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Pros –Online 3D min cut –Spatio temporal smooth cut Cons –Does not handle shadows –Ignore motion blur (LPF to avoid temporal aliasing) –Cannot separate translucent objects Summary

71 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Comparison Video Cutout Video object cut and paste Features 3D segmentation in 2 stages spatial-temporal manipulation 2D segmentation Frame base interface Graph nodes UI Performance 25 min 10 sec per Min cut 30 min 60 min 4-5 min 25 min ? 30 min Preprocessing Artist time Post processing Total

72 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Questions?

73 Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon The total energy: Foreground and background terms: Background terms: Link terms: Energy function

Download ppt "Advanced Topics in Computer Vision Spring 2006 Video Segmentation Tal Kramer, Shai Bagon Video Segmentation April 30 th, 2006."

Similar presentations

Change Detection C. Stauffer and W.E.L. Grimson, “Learning patterns of activity using real time tracking,” IEEE Trans. On PAMI, 22(8):747-757, Aug 2000.

Change Detection C. Stauffer and W.E.L. Grimson, “Learning patterns of activity using real time tracking,” IEEE Trans. On PAMI, 22(8): , Aug 2000.
SOFT SCISSORS: AN INTERACTIVE TOOL FOR REALTIME HIGH QUALITY MATTING International Conference on Computer Graphics and Interactive Techniques ACM SIGGRAPH.

SOFT SCISSORS: AN INTERACTIVE TOOL FOR REALTIME HIGH QUALITY MATTING International Conference on Computer Graphics and Interactive Techniques ACM SIGGRAPH.
Cutting Images: Graphs and Boundary Finding Computational Photography Derek Hoiem, University of Illinois 09/15/11 “The Double Secret”, Magritte.

Cutting Images: Graphs and Boundary Finding Computational Photography Derek Hoiem, University of Illinois 09/15/11 “The Double Secret”, Magritte.
Graph cut Chien-chi Chen.

Graph cut Chien-chi Chen.
Spatial-Temporal Consistency in Video Disparity Estimation ICASSP 2011 Ramsin Khoshabeh, Stanley H. Chan, Truong Q. Nguyen.

Spatial-Temporal Consistency in Video Disparity Estimation ICASSP 2011 Ramsin Khoshabeh, Stanley H. Chan, Truong Q. Nguyen.
I Images as graphs Fully-connected graph – node for every pixel – link between every pair of pixels, p,q – similarity w ij for each link j w ij c Source:

I Images as graphs Fully-connected graph – node for every pixel – link between every pair of pixels, p,q – similarity w ij for each link j w ij c Source:
1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen

1 Video Processing Lecture on the image part (8+9) Automatic Perception Volker Krüger Aalborg Media Lab Aalborg University Copenhagen
GrabCut Interactive Image (and Stereo) Segmentation Carsten Rother Vladimir Kolmogorov Andrew Blake Antonio Criminisi Geoffrey Cross [based on Siggraph.

GrabCut Interactive Image (and Stereo) Segmentation Carsten Rother Vladimir Kolmogorov Andrew Blake Antonio Criminisi Geoffrey Cross [based on Siggraph.
GrabCut Interactive Foreground Extraction using Iterated Graph Cuts Carsten Rother Vladimir Kolmogorov Andrew Blake Microsoft Research Cambridge-UK.

GrabCut Interactive Foreground Extraction using Iterated Graph Cuts Carsten Rother Vladimir Kolmogorov Andrew Blake Microsoft Research Cambridge-UK.
GrabCut Interactive Foreground Extraction using Iterated Graph Cuts Carsten Rother Vladimir Kolmogorov Andrew Blake Microsoft Research Cambridge-UK.

GrabCut Interactive Foreground Extraction using Iterated Graph Cuts Carsten Rother Vladimir Kolmogorov Andrew Blake Microsoft Research Cambridge-UK.
A Gimp Plugin that uses “GrabCut” to perform image segmentation

A Gimp Plugin that uses “GrabCut” to perform image segmentation
Graph-Based Image Segmentation

Graph-Based Image Segmentation
Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.

Stephen J. Guy 1. Photomontage Photomontage GrabCut – Interactive Foreground Extraction 1.
Human-Computer Interaction Human-Computer Interaction Segmentation Hanyang University Jong-Il Park.

Human-Computer Interaction Human-Computer Interaction Segmentation Hanyang University Jong-Il Park.
1 s-t Graph Cuts for Binary Energy Minimization  Now that we have an energy function, the big question is how do we minimize it? n Exhaustive search is.

1 s-t Graph Cuts for Binary Energy Minimization  Now that we have an energy function, the big question is how do we minimize it? n Exhaustive search is.
Graph-based image segmentation Václav Hlaváč Czech Technical University in Prague Faculty of Electrical Engineering Department of Cybernetics Prague, Czech.

Graph-based image segmentation Václav Hlaváč Czech Technical University in Prague Faculty of Electrical Engineering Department of Cybernetics Prague, Czech.
GrabCut Interactive Image (and Stereo) Segmentation Joon Jae Lee Keimyung University Welcome. I will present Grabcut – an Interactive tool for foreground.

GrabCut Interactive Image (and Stereo) Segmentation Joon Jae Lee Keimyung University Welcome. I will present Grabcut – an Interactive tool for foreground.
Authers : Yael Pritch Alex Rav-Acha Shmual Peleg. Presenting by Yossi Maimon.

Authers : Yael Pritch Alex Rav-Acha Shmual Peleg. Presenting by Yossi Maimon.
Image Segmentation some examples Zhiqiang wang zwang22@kent.edu.

Image Segmentation some examples Zhiqiang wang
Lecture 6 Image Segmentation

Lecture 6 Image Segmentation

Similar presentations

Ads by Google