1. Hebrew University Image Processing - 2009 1 Exercise Class 8 Panoramas – Stitching and Blending Min-Cut Stitching Many slides from Alexei Efros

2. Hebrew University Image Processing - 2009 2 Why Mosaic ? Are you getting the whole picture? Compact Camera FOV = 50 x 35°

3. Hebrew University Image Processing - 2009 3 Why Mosaic ? Are you getting the whole picture? Compact Camera FOV = 50 x 35° Human FOV = 200 x 135°

4. Hebrew University Image Processing - 2009 4 Why Mosaic ? Are you getting the whole picture? Compact Camera FOV = 50 x 35° Human FOV = 200 x 135° Panoramic Mosaic = 360 x 180°

5. Hebrew University Image Processing - 2009 5 Stages in building panoramas Find alignment between overlapping images Choose motion transformation between images (translation, translation + rotation, affine, homography) Choose compositing surface for warping Assign pixels in the panorama to source images Seamlessly blend images

6. Hebrew University Image Processing - 2009 6 Stitching the images together Why is this a challenge ? Exposure differences Scene illumination Blurring (miss-registration) Ghosting (moving objects) Goal – invisible seams between images Minimal amount of seams artifacts: edges that did not appear in the original images

7. Hebrew University Image Processing - 2009 7 Approaches Assuming that the images have already been aligned Simple seam location –Stitching Seam location + smoothing Blending the transition between images –Feathering (Alpha blending); Pyramid blending; Gradient domain blending Search for optimal seam –Dynamic programming; Graph cuts

8. Hebrew University Image Processing - 2009 Cut & Paste using Center Strips (Voronoi) 1 2 3 Strip taken from “2” Stitching The Panorama - Simple seam location Voronoi diagrams

9. Hebrew University Image Processing - 2009 9 Stitching The Panorama - Simple seam location t t+1 t+2

10. Hebrew University Image Processing - 2009 10 Image Blending

11. Hebrew University Image Processing - 2009 11 Feathering 0 1 0 1 + = I left (x,y) = (  R,  G,  B) I right (x,y) = ((1–α)R, (1–α)G, (1–α)B) I blend = I left + I right (α)(α)(1–α)

12. Hebrew University Image Processing - 2009 12 Affect of Window Size 0 1 left right 0 1

13. Hebrew University Image Processing - 2009 13 Affect of Window Size 0 1 0 1

14. Hebrew University Image Processing - 2009 14 Good Window Size 0 1 “Optimal” Window: smooth but not ghosted

15. Hebrew University Image Processing - 2009 15 Band-pass filtering Laplacian Pyramid (subband images) Created from Gaussian pyramid by subtraction Gaussian Pyramid (low-pass images)

16. Hebrew University Image Processing - 2009 16 Laplacian Pyramid Enables to Blend low frequencies Over a large spatial range And high frequencies over a short range

17. Hebrew University Image Processing - 2009 17 Pyramid Blending 0 1 0 1 0 1 Left pyramidRight pyramidblend

18. Hebrew University Image Processing - 2009 18 Gradient-domain Image Stitching The seam between the stitched images should be invisible. How this can be achieved ? 1.Compute the derivatives of the input images (I x1, I y1, I x2 I y2 ) 2.Blend the derivative images to form a field F = (Fx; Fy) 3.Find the mosaic image I’ whose gradients are closest to F (solve Poisson’s equation)

19. Hebrew University Image Processing - 2009 19 Gradient-domain Image Stitching (Levin, Zomet, Peleg, Weiss, 2004)

20. Hebrew University Image Processing - 2009 20 Poison Image Editing (Perez, Gangnet. Blake 2003)

21. Hebrew University Image Processing - 2009 21 Perez et al. 2003 Limitations: Colored backgrounds “bleed through” Images need to be very well aligned editing

22. Hebrew University Image Processing - 2009 22 What about moving objects ?

23. Hebrew University Image Processing - 2009 23 Don’t blend, CUT! (Search for optimal seam) So far we only tried to blend between two images. What about finding an optimal seam? Moving objects become ghosts

24. Hebrew University Image Processing - 2009 24 Where should the cut pass ?

25. Hebrew University Image Processing - 2009 25 Davis, 1998 Segment the mosaic Single source image per segment Avoid artifacts along boundries –Dijkstra’s algorithm

26. Hebrew University Image Processing - 2009 26 Input texture B1B2 Random placement of blocks block B1 B2 Neighboring blocks constrained by overlap B1B2 Minimal error boundary cut Another application for this approach: Texture synthesis (Efros & Freeman, 2001)

27. Hebrew University Image Processing - 2009 27 min. error boundary Minimal error boundary overlapping blocksvertical boundary _ = 2 overlap error

28. Hebrew University Image Processing - 2009 28

29. Hebrew University Image Processing - 2009 29

30. Hebrew University Image Processing - 2009 30 Maximum flow problem Max flow problem: Each edge is a “pipe” Find the largest flow F of “water” that can be sent from the “source” to the “sink” along the pipes Edge weights give the pipe’s capacity “source” A graph with two terminals S T “sink” a flow F

31. Hebrew University Image Processing - 2009 31 Minimum cut problem Min cut problem: Find the cheapest way to cut the edges so that the “source” is completely separated from the “sink” Edge weights now represent cutting “costs” a cut C “source” A graph with two terminals S T “sink”

32. Hebrew University Image Processing - 2009 32 Max flow/Min cut theorem Max Flow = Min Cut: Maximum flow saturates the edges along the minimum cut. Ford and Fulkerson, 1962 Problem reduction! Ford and Fulkerson gave first polynomial time algorithm for globally optimal solution “source” A graph with two terminals S T “sink”

33. Hebrew University Image Processing - 2009 33 How does min-cut relates to our problem ? Define the problem of finding the seam as a min-cut problem : the selected path will run between pairs of pixels.

34. Hebrew University Image Processing - 2009 34 What are the nodes of the graph location of pixels p=(x,y). What are the edges of the graph each pixel (node) has 4 neighbors pixel The weight of the edge (flow capacity) is the color difference between pairs of pixels that the edge connects W(p1, p2,A,B) = ||A(p1)−B(p1)||+||A(p2)−B(p2)|| p1,p2 are two adjacent pixels A(p1) and B(p1) be the pixel colors at the location p1 in image A and B, respectively What are the Source and Target of the graph Pixels we want to define explicitly from which image to take What is the meaning of the resulting cut The cut location will determine where do we want to have the seam between the images How does min-cut relates to our problem ?

35. Hebrew University Image Processing - 2009 35 Img B Edge Weights If the edge connects pixels from Image B that has same (or very close) colors of image A in both sides of the edges: Weight is very small -> we want this edge in the cut If the edge connects pixels from Image B that has different colors from image A in any side of the cut Weight is bigger (as the difference is bigger) -> we don’t want this edge in the cut Img A (x,y) (x+1,y) Img A (x,y)(x+1,y) Img B W(p 1,p 2,A,B) = ||A(p 1 )−B(p 1 )||+||A(p 2 )−B(p 2 )||

36. Hebrew University Image Processing - 2009 36 Applying Min cuts on images (simple example à la Boykov&Jolly, ICCV’01) n-links s t a cut hard constraint hard constraint Minimum cost cut can be computed in polynomial time (max-flow/min-cut algorithms) the selected path will run between pairs of pixels. Stitch from im1 Stitch from im2

37. Hebrew University Image Processing - 2009 37 Putting it all together Compositing images / mosaics: Simple seam location + smoothing transitions (blending) Feathering (Alpha blending) Pyramid blending Gradient domain blending –Less suitable when there is miss-alignment or moving objects Search for optimal seam Dynamic programming Graph cuts –Less suitable for thin strips –Less suitable when global differences (intensity) are presence

38. Hebrew University Image Processing - 2009 38 Stitching Example

39. Hebrew University Image Processing - 2009 39 Why do we need the min-cut ? Simple Stitchingmin-cut Simple Stitching min-cut

40. Hebrew University Image Processing - 2009 40 Mosaic Stitching Examples

41. Hebrew University Image Processing - 2009 41 Exam 1, Question 2 Goal: segment object from background

42. Hebrew University Image Processing - 2009 42 Exam 1, Question 2 Key observation: different gradient directions – Compute derivatives Ix, Iy – Calculate gradient direction: (modulo π ) – Binarize by a threshold

43. Hebrew University Image Processing - 2009 43 Exam 2 Date: Thursday 17/12 Q & A at Monday 14/12 Some hints: -Simple ideas -Try minimize heuristics -Nothing that you did not see -The principle is most important; however, you should roughly how do you implement it.