1. Image Stitching

2. Panoramic Image Mosaics
Full screen panoramas (cubic):
Mars:
2003 New Years Eve:
Richard Szeliski
Image Stitching

3. Gigapixel panoramas & images
Mapping / Tourism / WWT
Medical Imaging
Richard Szeliski
Image Stitching

4. Image Mosaics
… + =
Goal: Stitch together several images into a seamless composite
Richard Szeliski
Image Stitching

5. Today’s lecture Image alignment and stitching motion models
image warping
point-based alignment
complete mosaics (global alignment)
compositing and blending
ghost and parallax removal
Richard Szeliski
Image Stitching

6. Readings Szeliski, CVAA:
Chapter 3.6: Image warping
Chapter 5.1: Feature-based alignment (in preparation)
Chapter 9.1: Motion models
Chapter 9.2: Global alignment
Chapter 9.3: Compositing
Recognizing Panoramas, Brown & Lowe, ICCV’2003
Szeliski & Shum, SIGGRAPH'97
Richard Szeliski
Image Stitching

7. Motion models

8. Motion models
What happens when we take two images with a camera and try to align them?
translation?
rotation?
scale?
affine?
perspective?
VideoMosaic
Richard Szeliski
Image Stitching

9. Image Warping

10. Image Warping image filtering: change range of image g(x) = h(f(x))
image warping: change domain of image
g(x) = f(h(x)) f x f x h f x f x h
Richard Szeliski
Image Stitching

11. Image Warping image filtering: change range of image g(x) = h(f(x))
image warping: change domain of image
g(x) = f(h(x)) f g h f g h
Richard Szeliski
Image Stitching

12. Parametric (global) warping
Examples of parametric warps:
aspect
translation
rotation
perspective
affine
cylindrical
Richard Szeliski
Image Stitching

13. 2D coordinate transformations
Richard Szeliski
Image Stitching

14. 2D coordinate transformations
Translation Rotation + translation Similarity Affine
Richard Szeliski
Image Stitching

15. Homography
A: Projective – mapping between any two PPs with the same center of projection
rectangle should map to arbitrary quadrilateral
parallel lines aren’t
but must preserve straight lines
same as: project, rotate, reproject
called Homography
PP2 H p p’
PP1
To apply a homography H
Compute p’ = Hp (regular matrix multiply)
Convert p’ from homogeneous to image coordinates

16. 2D coordinate transformations
Projective transformation or Homography Normalized homogeneous
Homogeneous, only defined up to a scale
Richard Szeliski
Image Stitching

17. 2D coordinate transformations
Richard Szeliski
Image Stitching

18. Image Warping
Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))?
h(x) x x’
f(x)
g(x’)
Richard Szeliski
Image Stitching

19. Forward Warping
Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
What if pixel lands “between” two pixels?
h(x) x x’
f(x)
g(x’)
Richard Szeliski
Image Stitching

20. Forward Warping
Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
What if pixel lands “between” two pixels?
h(x) x x’
f(x)
g(x’)
Richard Szeliski
Image Stitching

21. Forward Warping Non-integer x’ Cracks and holes
Nearest integer coordinate
Aliasing
Jumping
Splatting
Distribute to 4 neighbors (weighted) and normalize later
Both aliasing and blur
Cracks and holes
Filling hole with nearby neighbors
Further aliasing and blurring

22. Inverse Warping
Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x)
What if pixel comes from “between” two pixels?
h(x) x x’
f(x)
g(x’)
Richard Szeliski
Image Stitching

23. Inverse Warping
Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x)
What if pixel comes from “between” two pixels?
Answer: resample color value from interpolated (prefiltered) source image x x’
f(x)
g(x’)
Richard Szeliski
Image Stitching

24. Interpolation Possible interpolation filters:
nearest neighbor
bilinear
bicubic (interpolating)
sinc / FIR
Needed to prevent “jaggies” and “texture crawl”
Richard Szeliski
Image Stitching

25. Interpolation Bicubic Bilinear ɑ: the derivative at x=1
Richard Szeliski
Image Stitching

26. Interpolation
Window sinc functions
Richard Szeliski
Image Stitching

27. Image Interpolation
All perform high-accuracy low-pass filtering while preserve details and avoid aliasing
Bilinear for speed
Bicubic and windowed sinc for better visual quality
Richard Szeliski
Image Stitching

28. Decimation
Interpolation: increase the image resolution Decimation (downsampling): reduce resolution
Samples are convolved with a low-pass filter before downsampled to avoid aliasing
Richard Szeliski
Image Stitching

29. Decimation Sample rate : same kernel as interpolation Commonly use r=2
Richard Szeliski
Image Stitching

30. Decimation
Frequency response
Richard Szeliski
Image Stitching

31. Decimation Linear filter gives a relatively poor response
Binomial filter cuts off a lot of frequencies but is useful for computer vision analysis pyramid
Cubic filter: a=-1 filter has a shaper fall-off than the a=-0.5 filter
Cosine-windowed sinc function
QMF-9 filter for wavelet denoising and aliases a fair amount
9/7 analysis filter form JPEG2000
Richard Szeliski
Image Stitching

32. Prefiltering
Essential for downsampling (decimation) to prevent aliasing
MIP-mapping [Williams’83]:
build pyramid (but what decimation filter?):
block averaging
Burt & Adelson (5-tap binomial)
7-tap wavelet-based filter (better)
trilinear interpolation
bilinear within each 2 adjacent levels
linear blend between levels (determined by pixel size)
Richard Szeliski
Image Stitching

33. Prefiltering
Essential for downsampling (decimation) to prevent aliasing
Other possibilities:
summed area tables
elliptically weighted Gaussians (EWA) [Heckbert’86]
Richard Szeliski
Image Stitching

34. Resmaple on interpolated signal
To avoid blurring or aliasing
Richard Szeliski
Image Stitching

35. Motion models (reprise)

36. Motion models Translation 2 unknowns Affine 6 unknowns Perspective
3D rotation
3 unknowns
Richard Szeliski
Image Stitching

37. Planar perspective motion
Simplest case:
Overlapping photographic prints, scanned pieces
Translation and rotation are usually adequate
Four unknowns
Richard Szeliski
Image Stitching

38. Planar perspective motion
Planar Homography
Richard Szeliski
Image Stitching

39. Planar Perspective Motion
Extract features from images
Match features using RANSAC and get inliers
Estimate a homography H given pairs of corresponding features
Least squares fitting
Richard Szeliski
Image Stitching

40. Solving for homography
p’ = Hp
Can set scale factor i=1. So, there are 8 unknowns.
Set up a system of linear equations:
Ah = b
where vector of unknowns h = [a,b,c,d,e,f,g,h]T
Need at least 8 eqs, but the more the better…
Solve for h. If overconstrained, solve using least-squares
Richard Szeliski
Image Stitching

41. Image warping with homographies
image plane in front
image plane below
Richard Szeliski
Image Stitching

42. Plane perspective mosaics
8-parameter generalization of affine motion
works for pure rotation or planar surfaces
Limitations:
local minima
slow convergence
difficult to control interactively
Richard Szeliski
Image Stitching

43. Rotational mosaics Directly optimize rotation and focal length
Advantages:
ability to build full-view panoramas
easier to control interactively
more stable and accurate estimates
GrandCanyon panorama
Richard Szeliski
Image Stitching

44. 3D → 2D Perspective Projectionu
(Xc,Yc,Zc) uc f
Richard Szeliski
Image Stitching

45. Rotational panoramas Setting With known f0, f1, solve R10
Richard Szeliski
Image Stitching

46. Rotational panoramas
More stable for large-scale image stitching algorithms than homography
How to solve the unknowns?
Update R10 with incremental rotation matrix R(w) to current estimate R10
Richard Szeliski
Image Stitching

47. Rotations and quaternions
How do we represent rotation matrices?
Axis / angle (n,θ) R = I + sinθ [n] + (1- cosθ) [n]2 (Rodriguez Formula), with [n] = cross product matrix (see paper)
Unit quaternions [Shoemake SIGG’85] q = (n sinθ/2, cosθ/2) = (w,s) quaternion multiplication (division is easy) q0 q1 = (s1w0+ s0w1, s0 s1-w0·w1)
Richard Szeliski
Image Stitching

48. Incremental rotation update
Small angle approximation ΔR = I + sinθ [n] + (1- cosθ) [n] ≈ θ [n] = [ω] linear in ω
Update original R matrix R ← R ΔR
Richard Szeliski
Image Stitching

49. Rotational panoramas
How to solve the unknowns of R10? Compute the incremental rotation vector w,
Richard Szeliski
Image Stitching

50. Rotational mosaic Projection equations Project from image to 3D ray
(x0,y0,z0) = (u0-uc,v0-vc,f)
Rotate the ray by camera motion
(x1,y1,z1) = R01 (x0,y0,z0)
Project back into new (source) image
(u1,v1) = (fx1/z1+uc,fy1/z1+vc)
Richard Szeliski
Image Stitching

51. Gap Closing
Estimate a series of rotation matrices and focal lengths to create large panoramas
Unfortunately, accumulated errors
Hard to estimate accurate focal length
Rarely produce a closed 360 pararoma
Richard Szeliski
Image Stitching

52. Gap Closing
Estimate a series of rotation matrices and focal lengths to create large panoramas
Unfortunately, accumulated errors
Rarely produce a closed 360 pararoma
Wrong focal length: f=510; correct focal length f=468
Richard Szeliski
Image Stitching

53. Gap Closing Matching the first image and the last one
Compute the gap angle
Distribute the gap angle evenly across the whole sequence
Modify rotations by
Update focal length
Only works for 1D panorama where the camera is continuously turning in the same direction
Richard Szeliski
Image Stitching

54. Application: Video Summarization and compression
Video summarization by Teodosio and Bender (1993)
Video compression and video indexing
More info at
Richard Szeliski
Image Stitching

55. Applications
Video stitching the background scene to create a single sprite image that can be transmitted and used to re-create the background in each frame (Lee, ge Chen, lung Bruce Lin et al. 1997)
Richard Szeliski
Image Stitching

56. Image reprojection The mosaic has a natural interpretation in 3D
mosaic PP
The mosaic has a natural interpretation in 3D
The images are reprojected onto a common plane
The mosaic is formed on this plane
Richard Szeliski
Image Stitching

57. Perspective & rotational motion
Solve 8x8 or 3x3 system (see papers for details), and iterate (non-linear)
Patch-based approximation:
break up image into patches (say 16x16)
accumulate 2x2 linear system in each (local translational assumption)
compose larger system from smaller 2x2 results [Shum & Szeliski, ICCV’98]
Richard Szeliski
Image Stitching

58. Image Mosaics (Stitching)
[Szeliski & Shum, SIGGRAPH’97]
[Szeliski, FnT CVCG, 2006]

59. Image Mosaics (stitching)
Blend together several overlapping images into one seamless mosaic (composite) … + =
Richard Szeliski
Image Stitching

60. Mosaics for Video Coding
Convert masked images into a background sprite for content-based coding =
Richard Szeliski
Image Stitching

61. Establishing correspondences
Direct method:
Use generalization of affine motion model [Szeliski & Shum ’97]
Feature-based method
Extract features, match, find consisten inliers [Lowe ICCV’99; Schmid ICCV’98, Brown&Lowe ICCV’2003]
Compute R from correspondences (absolute orientation)
Richard Szeliski
Image Stitching

62. Absolute orientation Given two sets of matching points, compute R
[Arun et al., PAMI 1987] [Horn et al., JOSA A 1988] Procrustes Algorithm [Golub & VanLoan]
Given two sets of matching points, compute R
pi’ = R pi 3D rays
A = Σi pi pi’T = Σi pi piT RT = U S VT = (U S UT) RT
VT = UT RT
R = V UT
Richard Szeliski
Image Stitching

63. Stitching demo
Richard Szeliski
Image Stitching

64. Panoramas What if you want a 360 field of view?
mosaic Projection Cylinder
Richard Szeliski
Image Stitching

65. Cylindrical panoramas
Steps
Reproject each image onto a cylinder
Blend
Output the resulting mosaic
Richard Szeliski
Image Stitching

66. Cylindrical Panoramas
Map image to cylindrical or spherical coordinates
need known focal length
Image 384x300
f = 180 (pixels)
f = 280
f = 380
Therefore, it is desirable if the global motion model we need to recover is translation, which has only two parameters. It turns out that for a pure panning motion, if we convert two images to their cylindrical maps with known focal length, the relationship between them can be represented by a translation.
Here is an example of how cylindrical maps are constructed with differently with f’s. Note how straight lines are curved.
Similarly, we can map an image to its longitude/latitude spherical coordinates as well if f is given.
Richard Szeliski
Image Stitching

67. Determining the focal length
Initialize from homography H (see text or [SzSh’97])
Use camera’s EXIF tags (approx.)
Use a tape measure
Ask your instructor 1m 4m
Richard Szeliski
Image Stitching

68. 3D → 2D Perspective Projectionu
(Xc,Yc,Zc) uc f
Richard Szeliski
Image Stitching

69. Cylindrical projection
Map 3D point (X,Y,Z) onto cylinder
unwrapped cylinder
Convert to cylindrical coordinates X Y Z
cylindrical image
Convert to cylindrical image coordinates
s defines size of the final image
unit cylinder
Richard Szeliski
Image Stitching

70. Cylindrical warping Given focal length f and image center (xc,yc) X YZ
(X,Y,Z)
(sinq,h,cosq)
Richard Szeliski
Image Stitching

71. Spherical warping φ cos φ sin φ cos θ cos φ
Given focal length f and image center (xc,yc) φ
(x,y,z)
cos φ
sin φ
(sinθcosφ,cosθcosφ,sinφ) Y Z
cos θ cos φ X
Richard Szeliski
Image Stitching

72. 3D rotation φ cos φ sin φ cos θ cos φ
Rotate image before placing on unrolled sphere φ
(x,y,z)
cos φ X Y Z
(sinθcosφ,cosθcosφ,sinφ)
sin φ
cos θ cos φ
p = R p
_ _
_ _
Richard Szeliski
Image Stitching

73. Radial distortion Correct for “bending” in wide field of view lenses
Richard Szeliski
Image Stitching

74. Fisheye lens Extreme “bending” in ultra-wide fields of view
Richard Szeliski
Image Stitching

75. Other types of mosaics
Can mosaic onto any surface if you know the geometry
See NASA’s Visible Earth project for some stunning earth mosaics
Richard Szeliski
Image Stitching

76. Slit images y-t slices of the video volume are known as slit images
take a single column of pixels from each input image
Richard Szeliski
Image Stitching

77. Slit images: cyclographs
Richard Szeliski
Image Stitching

78. Slit images: photofinish
Richard Szeliski
Image Stitching

79. Final thought: What is a “panorama”?
Tracking a subject
Repeated (best) shots
Multiple exposures
“Infer” what photographer wants?
Richard Szeliski
Image Stitching