Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume 27 Issue 3, August 2008 Presenter: Hon-Hang Chang Date: 2011/ 09/ 30 1

Outline Introduction Preliminaries Seam Carving using Graph Cut Forward Energy Results Conclusion 2

To treat video as 3D cube Introduction 1 D path on 2D image (Seam) 2D manifolds in a 3D volume In this paper the author define a new formulation of seam carving using graph cuts. Because of human perception, the temporal may even be more disturbing than spatial in video, as the human eye is highly sensitive to movement. 2D manifolds 3

Introduction A seam is a monotonic and connected path of pixels going from the top of the image to the bottom Or from left to right The image size is reduced by one either in the horizontal or the vertical dimension. video should support retargeting capabilities as it is displayed on TVs, computers, cellular phones. A whole seam 4

Preliminaries Among this, {It} t=1~N Extend the spatial L1-norm to a spatiotemporal L1-norm: Spatiotemporal L1-norm Spatial term Temporal term Where, α ∈ [0, 1] Motion artifacts are more noticeable Taking α= 0.3 5

Seam Carving using Graph Cuts S (Source) T (Sink) Source Sink arc1 arc2 arc3 arc4 Graph cut 6

Seam Carving using Graph Cuts An optimal seam must be: 1.1.Valid - A simple cut cannot define a seam carving 2. Monotonic - Including only one pixel in each row (or column) 3. Connected -The seam must be connected The proposed formulation extend seam carving to video and define a monotonic and connected 2D manifold seam inside the video cube. Optimal seam in graph cut 7

Seam Carving using Graph Cuts Graph cut for image (Non-monotonic) Every internal node P is connected to its four neighbors Nbr(P i, j ) = {Pi−1,j, Pi+1,j, Pi,j−1, Pi,j+1}. Pi,j Pi-1,j Pi,j-1 Pi,j+1 Pi+1,j Neighbors pixels ∂x(i, j) = |I(i, j + 1) − I(i, j)| ∂y(i, j) = |I(i+1, j ) − I(i, j)| Horizontal direction Vertical direction 8

Seam Carving using Graph Cuts Graph cut for image (Unconnected) Prove_1: 9

Seam Carving using Graph Cuts E1(i, j) =∂x(i, j) + ∂y(i, j) Graph cut for image (Original) Prove_2: 10

Seam Carving using Graph Cuts Graph cut for image We can use any energy function defined on the pixels as the weight of the forward horizontal arcs. Achieve the same results as the original dynamic programming based seam carving 11

Seam Carving using Graph Cuts Graph cut for video The extension to video is straightforward. 12

Seam Carving using Graph Cuts Graph cut for video Computation time is quadratic in the number of voxels. O(mn2) - n  #of nodes ; m  #of arcs Performance issues are encountered already for high resolution images The graph cut approach to seam carving allows us to extend the benefits of content-aware resizing to video. Still, the method is not perfect and no single energy function was shown to perform properly in all cases. Therefore, they introduce a new energy function that better protects media content, and improves video results. 13

Forward Energy Remove the seam with the least amount of energy from the image The inserted energy is due to new edges created by previously non adjacent pixels that become neighbors once the seam is removed To measure the real change in energy after a removal of a seam They measure the difference in the energy of the image after the removal (It=i+1) and the energy of only those parts that were not removed in the previous image 14

Forward Energy 15

+LR = | I(i, j+1)− I(i, j−1)| (arc weight between pi,j and pi,j+1) difference between the Left and Right neighbors +LU = | I(i-1, j)− I(i, j−1)| (arc weight between pi,j and pi-1,j) difference between the Left and Up neighbors -LU = | I(i+1, j)− I(i, j−1)| (arc weight between pi,j and pi+1,j) difference between the Left and Up neighbors with respect to the end point of the arrow Forward Energy Forward Energy in Graph Cut 16

Forward Energy Forward Energy in Graph Cut +LR = | I(i, j+1)− I(i, j−1)| (arc weight between pi,j and pi,j+1) difference between the Left and Right neighbors +LU = | I(i-1, j)− I(i, j−1)| (arc weight between pi,j and pi-1,j) difference between the Left and Up neighbors -LU = | I(i+1, j)− I(i, j−1)| (arc weight between pi,j and pi+1,j) difference between the Left and Up neighbors with respect to the end point of the arrow 17

Results 18

Results 19

Conclusion 20 Video retargeting is achieved using graph cuts and we have shown a construction that is consistent with the dynamic programming approach.