1. Image Processing Goal –Start simple: look at small windows –Identify useful image structures (‘Clues’ useful for recognizing objects) –Eliminate irrelevant aspects of image appearance (neglect appearance variations that don’t help to explain object identity) First step (usually) in any algorithm

2. We don’t “see” most information in image

3. + Can add invisible change

4. We don’t “see” most information in image + = Can add invisible change

5. Important information small fraction of total Initial Image

6. Important information small fraction of total Initial Image Important information Boundary locations boundary sharpness size/direction of brightness change

7. Important information small fraction of total Initial Image Important information Boundary locations edge sharpness; size/direction of brightness change Reconstruct good semblance of original image just using info at boundaries Reconstruction Elder: IJCV 34(2/3), 97–122 (1999)

8. Image Processing Goal –Small windows –Identify useful image structures (`clues’) –Eliminate irrelevant aspects Example 1 – Emphasize signal; suppress noise

9. Image Processing Goal –Small windows –Identify useful image structures (`clues’) –Eliminate irrelevant aspects Example 2 –Detect “boundaries” (jumps in brightness) –De-emphasize slow variations in brightness

10. Why look at small windows? Local redundancy –Neighboring image points strongly related (points on same object have similar color, texture)  Don’t need to know brightness at every pixel  Isolate just useful information (eg, average brightness in window) What’s behind the patch?

11. Why look at small windows? Details are important! Boundary signals presence of some object

12. Note All image locations are equal –Mannequin equally likely to be anywhere in image

13. Note All image locations are equal* –Mannequin equally likely to be anywhere in image *Not quite true for photographs (people center and compose images)

14. Conclusion Analyze image over every small window...

15. Conclusion Analyze image over every small window Easiest: linear weighted sum of brightness in each window...

16. Conclusion Analyze image over every small window Easiest: linear weighted sum of brightness in each window  Convolution (filtering)...

17. What is image filtering? For each pixel, modify value based on values of pixels nearby

18. Linear Filtering New pixel value = weighted sum of nearby pixels Uses: –Integrate information over regions –Clean up noisy images –Analyze image at different resolutions –Detect image patterns, brightness boundaries –Connects to Fourier analysis

19. Example (animated) 1D “image” Average Sum nearby pixels with weights Filtered Image I' Averaging makes filtered curve smoother Called mask, kernel, filter…

20. 2D Linear Filtering 1 1 1 1 1 1 1 1 1 3 x 3 filter (averaging or box filter) Image Note: computer vision filters usually have small number of pixels

21. Image 2D Linear Filtering Filtered Image

22. Image 2D Linear Filtering Filtered Image

23. Image 2D Linear Filtering Filtered Image

24. Image 2D Linear Filtering Filtered Image

25. Image 2D Linear Filtering Filtered Image

26. Image 2D Linear Filtering Filtered Image

27. Image 2D Linear Filtering Filtered Image

28. Image 2D Linear Filtering Filtered Image

29. Filtering Equations (correlation!) Original image filter (or mask, kernel…) Filtered image

30. Filtering Equations (correlation!) Original image filter Filtered image is a “little image” containing the weights with which the pixels of are summed Procedure For each filter position Multiply filter and image entries in corresponding positions Sum and record result at position under filter center

31. Filtering Equations (correlation!) Original image Filtered image filter

32. Filtering Equations (correlation!) Original image Filtered image filter Index ranges give filter size, here (2N+1) x (2N+1)

33. Filtering Equations (correlation!) Original image Filtered image Easiest: use odd sized filters, symmetric index range [–N,N] so filter center at (0,0) Index ranges give filter size, here (2N+1) x (2N+1) filter

34. One Dimension (correlation!) Filtered image For filter of size (2N+1) centered on (0,0)

35. Convolution Like Correlation with Filter Reversed 1D 2D

36. Convolution Like Correlation with Filter Reversed 1D 2D ‘-’ instead of ‘+’ crucial change!

37. Convolution Like Correlation with Filter Reversed Many nice properties-a kind of multiplication 1D 2D

38. Convolution From now on, linear filtering equals convolution (unless I say otherwise)

39. Convolution procedure Original image filter Convolved image Procedure For each filter position Flip (reflect) filter in both x and y directions Multiply filter and image entries in corresponding positions Sum and record result at position under filter center

40. Convolution: symmetric form (doesn’t work for correlation) 1D

41. Convolution: symmetric form (doesn’t work for correlation) 1D {

42. Convolution: symmetric form (doesn’t work for correlation) 1D {

43. Convolution: symmetric form (doesn’t work for correlation) 1D Convention: extend filter. Assign W(i)=0 for out-of-range

44. Convolution: symmetric form (doesn’t work for correlation) 1D Convention: extend filter. Assign W(i)=0 for out-of-range

45. Convolution: symmetric form (doesn’t work for correlation) 1D This extends for any number of convolutions (again, with convention that everything out of range is zero for W and V)

46. Convolution: symmetric form (doesn’t work for correlation) 2D

47. Convolution: symmetric form (doesn’t work for correlation) 2D

48. Convolution like multiplication! Commutative Associative Distributive (linear)

49. Convolution like multiplication! Commutative

50. Convolution like multiplication! Commutative

51. Convolution like multiplication! Associative

52. 0 Convolution: More properties Shift invariance 0 1 1 0 1 0 0 0 Compare 1 0 0 1 0 1 0 0 0 vs

53. Convolution: More properties Shift invariance  Fourier Transform connection  Convolution equivalent* to multiplication after Fourier Transform!!

54. Some Examples Filter mask 1.0 Filtered image

55. Some Examples Filter mask 1.0 Plot of mask (weights) Filtered image

56. Some Examples Filter mask 1.0 Filtered image

57. Some Examples In 1D the plot of W would look like this. Filter mask Filtered image

58. Filter mask 1.0

59. Filter mask 1.0

60. Next example

61. Filter mask 1.0 Shifted from (0,0)

62. Filter mask 1.0

63. Filter mask 1.0

64. Filter mask New example

65. Filter mask 1 1 1 1 1 1 1 1 1

66. Averaging filter  Blur

67. OriginalFiltered (1D) Filter mask

68. OriginalFiltered (1D) Filter mask

69. OriginalFiltered (1D) Filter mask

70. OriginalFiltered (1D) Filter mask

71. OriginalFiltered (1D) Filter mask Note how the original sharp transition gets blurred

72. 2.0 1.0 Filter mask Warm up…

73. 1 Filter mask 2 1 - Equivalent

74. 2.0 1.0 Filter mask Warm up…

75. 2.0 Filter mask

77. 2.0 Filter mask -

78. 1.89 (a peak in a trough)

80. Original in 1D - 2.0 Filter mask

81. Original in 1D Filter mask

82. Original in 1D Filter mask

83. Original in 1D Filter mask

84. Original in 1D Filter mask

86. 0 1 1 0 1 0 ? Different example

87. 0 1 1 0 1 0 Different example

88. 1D example 1 =? o

89. 1D example o 1 = Only the jump survives!

90. A Detail: how to deal with border

91. Dealing with image border What happens here? Trying to sum over pixels outside the image

92. Dealing with image border Various choices: 1) Pad with zeros… 0 0 0 0 0 0 0 0 0 0

93. Dealing with image border Various choices: 1)Pad with zeros 2)Duplicate border

94. Dealing with image border Various choices: 1)Pad with zeros 2)Duplicate border

95. Dealing with image border Various choices: 1)Pad with zeros 2)Duplicate border 3)Wrap (not so important in this class)

96. Dealing with image border Various choices: 1) Pad with zeros 2) Duplicate border 3) Wrap 4) Or just crop (don’t try to extend original image; filtered image is smaller than original) Image Filtered Image

97. Filtering to reduce noise Noise = what we don’t care about –Assume random noise added at each pixel –Reduce noise by averaging over windows Random noise from different pixels tends to cancel Signal not much affected (image is redundant---pixel’s neighbors have similar brightness)

98. Simple Additive Noise

99. ( Image = signal + random noise) Assume –No dependence of noise size on signal –Expected value of noise is zero –Noise added at each pixel independently –Type of noise is the same at all pixels. or, more precisely:

100. Averaging filter to reduce noise

101. Averaging Filter: Definition Mask has positive weights summing to 1 Replaces each pixel with weighted average over its neighborhood Example: BOX filter has all weights equal 1 1 1 1 11 1 1 1 1/9

102. Averaging several times...

103. Image gets smoother; noise in patches instead of speckles

104. For smooth signals, averaging doesn’t affect the signal much and improves Signal/Noise Example: Image = Constant + Noise –Average image over n-pixel window

105. For smooth signals, averaging doesn’t affect the signal much and improves Signal/Noise Example: Image = Constant + Noise –Average image over n-pixel window Smaller because of cancellations

106. Averaging a noisy image You can copy this code to matlab ( substitute your image for prowler.pgm) I=imread('prowler.pgm'); imagesc(I), figure(gcf ) % “,” separates commands, figure(gcf) brings current figure to front (gcf ==“get current figure”) colormap gray % shows image as black and white S= size(I); % size of image I In=double(I)+randn(S(1),S(2))*50; % Adds noise; randn creats a matrix of the given size with Gaussian random entries N(0,1) % `double’ forces image I to type double imagesc(In),figure(gcf) Is= conv2(double(In),ones(3,3)/9);imagesc(Is),figure(gcf) % convolve image with box filter. ``ones” creates a matrix % of given size containing ones Is= conv2(double(Is),ones(3,3)/9);imagesc(Is),figure(gcf) pause % Wait for keyboard input before going on % what does averaging again do? REPEAT Is= conv2(double(Is),ones(3,3)/9);imagesc(Is),figure(gcf), pause close all %close all figure displays

107. Averaging reduces noise k=100; T=1000; % k= image length (for 1D noise image) % T = number of trials RandIm =randn(k,1); plot(RandIm), figure(1), title('1D Noise Image') AverageNoise = sum((1/k)*RandIm), pause figure(2) hist(sum((1/k)*randn(k,T))); axis([-1 1 0 inf]); xlabel([int2str(T),' trials of noise averaged over ', int2str(k),' pixels']); title('histogram')

108. Averaging for a normal image... (no noise)

109. What else is averaging good for? Remember original goal of image processing –Identify useful image structures (`clues to objects’) –Throw away less useful parts of image

110. What else is averaging good for? Remember original goal of image processing –Identify useful image structures (`clues to objects’) –Throw away less useful parts of image After smoothing… – Each pixel contains average brightness in its neighborhood. –This may be all you need to know about the neighborhood  To save only important information, keep just one pixel per neighborhood.

111. Original Smoothed with 1 1 1 1 1 1 1 1 1 Keeping only every other pixel (along x and y directions) New image ¼ size All important information preserved

112. Image Compression Original Subsampled (every 7 th pixel)

113. Image Compression Original Subsampled (every 7 th pixel) Smoothed, then sampled

114. Averaging: problems

115. Example: Averaging/smoothing with Box Filter Star Star “averaged” with Box

116. Example: Averaging/smoothing with Box Filter Star Star “averaged” with Box  Not very smooth!

117. Example: Averaging/smoothing with Box Filter

118. Artifact from sharp edge of box filter

119. Improved averaging filter

120. Smoothing as Inference of Signal True signal is smooth. To infer a pixel’s “true” brightness without noise, look at brightness of nearby pixels Signal + Noise signal plus noise

121. Smoothing as Inference of Signal Infer brightness at central pixel only using nearby pixels Signal + Noise signal plus noise Mask matched to signal

122. Smoothing as Inference of Signal Infer brightness at central pixel only using nearby pixels  Adjust size of averaging mask (Match signal smoothness: reduce noise without oversmoothing signal) Signal + Noise signal plus noise Mask matched to signal

123. Smoothing as Inference of Signal Mask size should match signal smoothness Signal signal plus noise Mask + Noise

124. Smoothing as Inference of Signal mask size should match signal smoothness Closer pixels are more similar Similarity falls off smoothly with distance Signal + Noise signal plus noise mask

125. Smoothing as Inference of Signal mask size should match signal smoothness Closer pixels are more similar Similarity falls off smoothly with distance  Make mask weights decrease smoothly with distance from filter center Signal + Noise signal plus noise mask

126. From box filter to Gaussian

127. Averaging with Gaussian mask Gaussian weights nearby pixels more Smooth roll off in weights

128. Gaussian Smoother (Rotationally Symmetric) Gaussian filter gives reasonable model of blurring (eg from lens)

129. Star Smoothing with Gaussian Blurry Star (Gaussian Smoothed)

130. Smoothing with Gaussian

131. Box Smoothing Gaussian Smoothing Artifact from sharpness of box filter No artifact

132. Gaussian Filter: useful property Convolving 2 Gaussians yields* new Gaussian! Recall Gaussian Definition:

133. Gaussian Filter: useful property Convolving 2 Gaussians yields* new Gaussian! Recall Gaussian Definition: New Gaussian is wider, with (more blur)

134. Gaussian Filter: useful property Convolving 2 Gaussians yields* new Gaussian! Recall Gaussian Definition: New Gaussian is wider, with (more blur) * Strictly true only for continuous functions, not discrete filters defined on pixels

135. Gaussian Filter: useful property Convolving 2 Gaussians yields* new Gaussian! Recall Gaussian Definition: New Gaussian is wider, with (more blur) * Strictly true only for continuous functions, not discrete filters defined on pixels See later for equations

136. Gaussian Filter Associativity of convolution implies Several filterings with small Gaussians equivalent to single filtering with larger Gaussian

137. This generalizes… Convolving two smoothing filters gives a smoothing filter with more blur. Can get a lot of smoothing by convolving many times with a small (low blur) filter This is faster to compute…

138. Gaussian Filter and Scale Space Want to consider image at different resolutions (levels of detail, different scales)  Smooth image with different sized smoothers Branches,leaves Bushes Tree, ground, sky foreground, background

139. Gaussian Filter and Scale Space Want to consider image at different resolutions (levels of detail, different scales)  Smooth image with different sized smoothers Branches,leaves Bushes Tree, ground, sky foreground, background Human vision does this! (double face)

140. Gaussian Filter and Scale Space Want to consider image at different resolutions (levels of detail, different scales)  Smooth image with different sized smoothers Scale Space = same image viewed at different resolutions (size scales) Branches,leaves Bushes Tree, ground, sky foreground, background

141. Gaussian Filter and Scale Space Want to consider image at different resolutions (levels of detail, different scales)  Smooth image with different sized smoothers Scale Space = same image viewed at different resolutions (size scales) With Gaussians, only resolution matters… Whatever the sequence of Gaussian smoothing operations applied to an image, the result is the same as smoothing once with a single Gaussian of the appropriate size/resolution. (With other filters, two different series of smoothing operations can give different results at the same resolution) Branches,leaves Bushes Tree, ground, sky foreground, background

142. Gaussian Filter and Scale Space Scale Space (same image viewed at different resolutions) Many applications! –Fast image download ( Transmit low resolution, progressively upgrade to higher resolution) –Image editing/blending –Fast image searching ( search image first at low resolution, refine at higher resolution) –Image compression ….

143. Example: Image Blending

144. Implementation notes Gaussian has infinite size (G(x)>0 for all x) For efficiency, cut off filter at large x. What x? Rule of thumb: Use window with each side

145. Implementation note 2 Gaussian and BOX filters are separable

146. Implementation note 2 Gaussian and BOX filters are separable

147. Implementation note 2 Gaussian and BOX filters are separable  Can replace 2D convolution with two 1D ones: Much faster! 1. Convolve each row with 1D filter 2.Then convolve each column with 1D filter Reduces computation by factor width of filter

148. Image Separable Filtering (animated!) Filtered Image =

149. Image Separable Filtering (animated!) Filtered Image =

150. Aside: convolution for continuous functions discrete continuous

151. Aside: Continuous convolution Discrete (closer analog) continuous

152. Constants normalize the “weights” to 1, so Gaussian is an averaging filter. Above is for continuous functions. When you approximate Gaussian on grid of pixels, normalize so Note: normalization

153. Convolving 2 Gaussians See next page for equations for 1D convolution

154. Equations (Gaussian convolution in 1D) Completing square in exponent Change variable From normalization of Gaussian

155. Summary Filtering: analyze small image windows Linear filtering and correlation/convolution –Uses Smoothing/averaging/blurring Sharpening Boundary (edge) detection Next set of slides: pattern detection –Convolution: Technicalities Symmetric form Commutative, associative, etc. Dealing with boundary

156. Summary Smoothing –Uses Noise reduction (not so important for this class) Compression Scale space: analyze image at different resolutions (see later) Edge detection (see later) –Technicalities Gaussians (special nice properties) Normalize sum of weights to 1! Size of Gaussian mask (note: smaller gives faster computation) Separability Blurring a little many times  blurring a lot once