1. Image Filtering Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 02/02/10

2. Questions about HW 1?

3. Questions about class? Room change starting thursday: Everitt 163, same time

4. Key ideas from last class Lighting – Ambiguity between light source and albedo – Shading is a strong cue for shape – Interreflections, multiple sources, ambient light, etc. make lighting and shadows complicated Color constancy – Color can be rebalanced by making assumptions (e.g., average pixel is gray)

5. Lightness Perception from Ted Adelson

6. Lightness Perception from Ted Adelson

7. By nickwheeleroz, on Flickr

9. Karsch et al. in review

10. Today’s class How can we represent color? What is image filtering and how do we do it? What are some useful filters and what do they do? What is linear separability? Thinking in the frequency domain

11. Next two classes Today – Brief overview of color models – Image filtering Thursday – Images as frequencies – Image pyramids – Texture

12. Color spaces How can we represent color? http://en.wikipedia.org/wiki/File:RGB_illumination.jpg

13. Color spaces: RGB 0,1,0 0,0,1 1,0,0 Image from: http://en.wikipedia.org/wiki/File:RGB_color_solid_cube.png Some drawbacks Strongly correlated channels Non-perceptual

14. Color spaces: HSV

15. Color spaces: L*a*b* “Perceptually uniform” color space

16. If you had to choose, would you rather go without luminance or chrominance?

18. Most information in intensity Only color shown – constant intensity

19. Most information in intensity Only intensity shown – constant color

20. Most information in intensity Original image

21. The raster image (pixel matrix)

22. 0.920.930.940.970.620.370.850.970.930.920.99 0.950.890.820.890.560.310.750.920.810.950.91 0.890.720.510.550.510.420.570.410.490.910.92 0.960.950.880.940.560.460.910.870.900.970.95 0.710.81 0.870.570.370.800.880.890.790.85 0.490.620.600.580.500.600.580.500.610.450.33 0.860.840.740.580.510.390.730.920.910.490.74 0.960.670.540.850.480.370.880.900.940.820.93 0.690.490.560.660.430.420.770.730.710.900.99 0.790.730.900.670.330.610.690.790.730.930.97 0.910.940.890.490.410.78 0.770.890.990.93

23. Image filtering Image filtering: compute function of local neighborhood at each position Why bother? – Modify images Denoise, resize, enhance contrast, etc. – Extract information from images Edge detection, matching, find distinctive points, etc.

24. 111 111 111 Slide credit: David Lowe (UBC) Example: box filter

25. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0 0000000000 0000000000 000 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Credit: S. Seitz Image filtering 111 111 111

26. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 010 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Image filtering 111 111 111 Credit: S. Seitz

27. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 01020 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Image filtering 111 111 111 Credit: S. Seitz

28. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0102030 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Image filtering 111 111 111 Credit: S. Seitz

29. 0102030 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Image filtering 111 111 111 Credit: S. Seitz

30. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0102030 2010 0204060 4020 0306090 6030 0 5080 906030 0 5080 906030 0203050 604020 102030 2010 00000 Image filtering 111 111 111 Credit: S. Seitz Q?

31. What does it do? Replaces each pixel with an average of its neighborhood Achieve smoothing effect (remove sharp features) 111 111 111 Slide credit: David Lowe (UBC) Box Filter

32. Smoothing with box filter

33. Practice with linear filters 000 010 000 Original ? Source: D. Lowe

34. Practice with linear filters 000 010 000 Original Filtered (no change) Source: D. Lowe

35. Practice with linear filters 000 100 000 Original ? Source: D. Lowe

36. Practice with linear filters 000 100 000 Original Shifted left By 1 pixel Source: D. Lowe

37. Practice with linear filters Original 111 111 111 000 020 000 - ? (Note that filter sums to 1) Source: D. Lowe

38. Practice with linear filters Original 111 111 111 000 020 000 - Sharpening filter - Accentuates differences with local average Source: D. Lowe

39. Sharpening Source: D. Lowe

40. Other filters 01 -202 01 Vertical Edge (absolute value) Sobel

41. Other filters -2 000 121 Horizontal Edge (absolute value) Sobel Q?

42. Filtering vs. Convolution 2d filtering –h=filter2(f,g); or h=imfilter(f,g); 2d convolution –h=conv2(f,g);

43. Key properties of linear filters Linearity: filter(f 1 + f 2 ) = filter(f 1 ) + filter(f 2 ) Shift invariance: same behavior regardless of pixel location: filter(shift(f)) = shift(filter(f)) Any linear shift-invariant operator can be represented as a convolution Source: S. Lazebnik

44. More properties Commutative: a * b = b * a – Conceptually no difference between filter and signal Associative: a * (b * c) = (a * b) * c – Often apply several filters one after another: (((a * b 1 ) * b 2 ) * b 3 ) – This is equivalent to applying one filter: a * (b 1 * b 2 * b 3 ) Distributes over addition: a * (b + c) = (a * b) + (a * c) Scalars factor out: ka * b = a * kb = k (a * b) Identity: unit impulse e = [0, 0, 1, 0, 0], a * e = a Source: S. Lazebnik

45. Weight contributions of neighboring pixels by nearness 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003 5 x 5,  = 1 Slide credit: Christopher Rasmussen Important filter: Gaussian

46. Gaussian filters Remove “high-frequency” components from the image (low-pass filter) Convolution with self is another Gaussian – So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have – Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ√2 Separable kernel – Factors into product of two 1D Gaussians Source: K. Grauman

47. Separability of the Gaussian filter Source: D. Lowe

48. Separability example * * = = 2D convolution (center location only) Source: K. Grauman The filter factors into a product of 1D filters: Perform convolution along rows: Followed by convolution along the remaining column:

49. Separability Why is separability useful in practice?

50. Smoothing with Gaussian filter

51. Smoothing with box filter

52. Thinking in terms of frequency

53. Intensity Image Fourier Image http://sharp.bu.edu/~slehar/fourier/fourier.html#filtering

54. Signals can be composed += http://sharp.bu.edu/~slehar/fourier/fourier.html#filtering More: http://www.cs.unm.edu/~brayer/vision/fourier.html

55. Fourier Transform Fourier transform is in terms of frequency magnitude and phase Phase encodes spatial information (indirectly) Can convert back and forth losslessly To filter, multiply the Fourier images

56. Fourier Matlab demo

57. Some practical matters

58. How big should the filter be? Values at edges should be near zero Rule of thumb for Gaussian: set filter half-width to about 3 σ Practical matters

59. What is the size of the output? MATLAB: filter2(g, f, shape) – shape = ‘full’: output size is sum of sizes of f and g – shape = ‘same’: output size is same as f – shape = ‘valid’: output size is difference of sizes of f and g f gg gg f gg g g f gg gg full samevalid Source: S. Lazebnik

60. Practical matters What about near the edge? – the filter window falls off the edge of the image – need to extrapolate – methods: clip filter (black) wrap around copy edge reflect across edge Source: S. Marschner Q?

61. Practical matters – methods (MATLAB): clip filter (black): imfilter(f, g, 0) wrap around:imfilter(f, g, ‘circular’) copy edge: imfilter(f, g, ‘replicate’) reflect across edge: imfilter(f, g, ‘symmetric’) Source: S. Marschner

62. Things to remember Several options for color spaces Linear filtering is sum of dot product at each position Sometimes useful to think of images/filters in frequency domain Careful around edges 111 111 111

63. Next class Applications of filters Using filters for denoising and downsampling Using filters for matching Image pyramids, filter banks, and texture

64. Questions