Image Processing and Morphing Vision for Graphics CSE 590SS, Winter 2001 Richard Szeliski

Image Pyramids

1/8/2001Vision for Graphics3 Image Pyramids

1/8/2001Vision for Graphics4 Pyramid Creation “Laplacian” Pyramid Created from Gaussian pyramid by subtraction L l = G l – expand(G l+1 ) filter mask “Gaussian” Pyramid

1/8/2001Vision for Graphics5 Octaves in the Spatial Domain Bandpass Images Lowpass Images

1/8/2001Vision for Graphics6 Pyramids Advantages of pyramids Faster than Fourier transform Avoids “ringing” artifacts Many applications small images faster to process good for multiresolution processing compression progressive transmission Known as “mip-maps” in graphics community Precursor to wavelets Wavelets also have these advantages

1/8/2001Vision for Graphics7 laplacian level 4 laplacian level 2 laplacian level 0 left pyramidright pyramidblended pyramid

1/8/2001Vision for Graphics8 Pyramid Blending

Image Warping

1/8/2001Vision for Graphics10 Image Warping image filtering: change range of image g(x) = h(f(x)) image warping: change domain of image g(x) = f(h(x)) f x h f x f x h f x

1/8/2001Vision for Graphics11 Image Warping image filtering: change range of image g(x) = h(f(x)) image warping: change domain of image g(x) = f(h(x)) hh f f g g

1/8/2001Vision for Graphics12 Parametric (global) warping Examples of parametric warps: translation rotation aspect affine perspective cylindrical

1/8/2001Vision for Graphics13 2D coordinate transformations translation:x’ = x + t x = (x,y) rotation:x’ = R x + t similarity:x’ = s R x + t affine:x’ = A x + t perspective:x’  H x x = (x,y,1) (x is a homogeneous coordinate) These all form a nested group (closed w/ inv.)

1/8/2001Vision for Graphics14 Image Warping Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))? f(x)f(x)g(x’) xx’ h(x)h(x)

1/8/2001Vision for Graphics15 Forward Warping Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’) f(x)f(x)g(x’) xx’ h(x)h(x) What if pixel lands “between” two pixels?

1/8/2001Vision for Graphics16 Forward Warping Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’) f(x)f(x)g(x’) xx’ h(x)h(x) What if pixel lands “between” two pixels? Answer: add “contribution” to several pixels, normalize later (splatting)

1/8/2001Vision for Graphics17 Inverse Warping Get each pixel g(x’) from its corresponding location x = h -1 (x’) in f(x) f(x)f(x)g(x’) xx’ h -1 (x’) What if pixel comes from “between” two pixels?

1/8/2001Vision for Graphics18 Inverse Warping Get each pixel g(x’) from its corresponding location x = h -1 (x’) in f(x) What if pixel comes from “between” two pixels? Answer: resample color value from interpolated (prefiltered) source image f(x)f(x)g(x’) xx’

1/8/2001Vision for Graphics19 Interpolation Possible interpolation filters: nearest neighbor bilinear bicubic (interpolating) sinc / FIR Needed to prevent “jaggies” and “texture crawl” (see demo)

1/8/2001Vision for Graphics20 Prefiltering Essential for downsampling (decimation) to prevent aliasing MIP-mapping [Williams’83]: 1.build pyramid (but what decimation filter?): –block averaging –Burt & Adelson (5-tap binomial) –7-tap wavelet-based filter (better) 2. trilinear interpolation –bilinear within each 2 adjacent levels –linear blend between levels (determined by pixel size)

1/8/2001Vision for Graphics21 Prefiltering Essential for downsampling (decimation) to prevent aliasing Other possibilities: summed area tables elliptically weighted Gaussians (EWA) [Heckbert’86]

1/8/2001Vision for Graphics22 Image Warping – non-parametric Specify more detailed warp function Examples: splines triangles optical flow (per-pixel motion)

1/8/2001Vision for Graphics23 Image Warping – non-parametric Move control points to specify spline warp

Image Morphing

1/8/2001Vision for Graphics25 Image Morphing How can we in-between two images? 1.Cross-dissolve (all examples from [Gomes et al.’99])

1/8/2001Vision for Graphics26 Image Morphing How can we in-between two images? 2.Warp then cross-dissolve = morph

1/8/2001Vision for Graphics27 Warp specification How can we specify the warp? 1.Specify corresponding points interpolate to a complete warping function Nielson, Scattered Data Modeling, IEEE CG&A’93]

1/8/2001Vision for Graphics28 Warp specification How can we specify the warp? 2.Specify corresponding vectors interpolate to a complete warping function

1/8/2001Vision for Graphics29 Warp specification How can we specify the warp? 2.Specify corresponding vectors interpolate [Beier & Neely, SIGGRAPH’92]

1/8/2001Vision for Graphics30 Warp specification How can we specify the warp? 3.Specify corresponding spline control points interpolate to a complete warping function

1/8/2001Vision for Graphics31 Final Morph Result

Image Mosaics

1/8/2001Vision for Graphics33 Image Mosaics (stitching) Blend together several overlapping images into one seamless mosaic (composite) + + … +=

Mosaics for Video Coding Convert masked images into a background sprite for content-based coding =

1/8/2001Vision for Graphics35 f = 180 (pixels) Cylindrical Panoramas Map image to cylindrical or spherical coordinates need known focal length Image 384x300f = 380f = 280

1/8/2001Vision for Graphics36 Cylindrical warping Given focal length f and image center (x c,y c ) X Y Z (X,Y,Z)(X,Y,Z) (sin ,h,cos  )

1/8/2001Vision for Graphics37 Radial distortion Correct for “bending” in wide field of view lenses

1/8/2001Vision for Graphics38 Image Stitching 1.Align the images over each other 2.Blend the images together (demo)

1/8/2001Vision for Graphics39 Image Stitching Assignment 1.Take pictures on a tripod (or handheld) 2.Warp to cylindrical coordinates 3.Automatically compute pairwise alignments 4.Fix up the end-to-end alignment 5.Blend the images together 6.Crop the result and import into a viewer

1/8/2001Vision for Graphics40 Image registration (preview) How do we determine alignment between images? One possible answer: block matching (correlation), i.e., find minimum squared error

1/8/2001Vision for Graphics41 Image registration (preview) How do we determine alignment between images? Another possible answer: Fourier-domain correlation [Brown’92] or phase correlation [Kuglin & Hines’75] More on alignment/registration on Wednesday [Anandan]

1/8/2001Vision for Graphics42 Full-view Panorama

1/8/2001Vision for Graphics43 Texture Mapped Model

Image Enhancement

1/8/2001Vision for Graphics45 Image Enhancement Noise removal: low-pass filtering g(x) = h(x) f(x) =  i h(-i) f(x+i) median filtering anisotropic diffusion –iteratively smooth with similar neighbors [Perona & Malik’90] Sharpening: “unsharp masking” –remove a little bit of blurred image from original (darkroom trick)

1/8/2001Vision for Graphics46 Image Enhancement Super-resolution: hallucinate more detail from single low-res image [Freeman & Pasztor,ICCV’99] –“learn” mapping between levels combine several low-res images using known (linear) formation model [Irani’91;Mann’94]

1/8/2001Vision for Graphics47 Image Enhancement Super-resolution: [Freeman & Pasztor,ICCV’99] –mapping between levels depends on “training data”

1/8/2001Vision for Graphics48 Image Enhancement Brightness and gamma correction linearize input  brightness Histogram equalization balance (“flatten”) distribution of brightness values

1/8/2001Vision for Graphics49 Image Enhancement High dynamic range photography [Debevec et al.’97; Mitsunaga & Nayar’99] combine several different exposures together

1/8/2001Vision for Graphics50 Feature extraction edge detection zero crossings hysteresis filtering (Canny) orientationally selective filters edge linking polygonal simplification for line extraction

1/8/2001Vision for Graphics51 Feature extraction line detection Hough transform: the bad way (vote for all possible lines) Hough transform: better way (vote according to local orientation); can also extend to vanishing point voting VideoMouse application? (related to project 2)

1/8/2001Vision for Graphics52 Bibliography L. Williams. Pyramidal parametrics. Computer Graphics, 17(3):1--11, July L. G. Brown. A survey of image registration techniques. Computing Surveys, 24(4): , December C. D. Kuglin and D. C. Hines. The phase correlation image alignment method. In IEEE 1975 Conference on Cybernetics and Society, pages , New York, September J. Gomes, L. Darsa, B. Costa, and L. Velho. Warping and Morphing of Graphical Objects. Morgan Kaufmann Publishers, San Francisco Altos, California, G. M. Nielson. Scattered data modeling. IEEE Computer Graphics and Applications, 13(1):60--70, January T. Beier and S. Neely. Feature-based image metamorphosis. Computer Graphics (SIGGRAPH'92), 26(2):35--42, July 1992.

1/8/2001Vision for Graphics53 Bibliography L. Williams. Performace driven facial animation. Computer Graphics, 24(4): , P. Perona and J. Malik. Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(7): , July W. T. Freeman and E. C. Pasztor. Learning low-level vision. In Seventh International Conference on Computer Vision (ICCV'99), pages , Kerkyra, Greece, September M. Irani and S. Peleg. Improving resolution by image registration. Graphical Models and Image Processing, 53(3): , May S. Mann and R. W. Picard. Virtual bellows: Constructing high-quality images from video. In First IEEE International Conference on Image Processing (ICIP-94), volume I, pages , Austin, Texas, November 1994.

1/8/2001Vision for Graphics54 Bibliography Paul E. Debevec and Jitendra Malik. Recovering high dynamic range radiance maps from photographs. Proceedings of SIGGRAPH 97, pages , August ISBN Held in Los Angeles, California. T. Mitsunaga and S. K. Nayar. Radiometric self calibration. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'99), volume 1, pages , Fort Collins, June 1999.