1. Lecture 1: Images and image filtering CS4670/5670: Intro to Computer Vision Kavita Bala Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htmhttp://cvcl.mit.edu/hybridimage.htm

2. Lecture 1: Images and image filtering CS4670: Computer Vision Kavita Bala Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htmhttp://cvcl.mit.edu/hybridimage.htm

3. Lecture 1: Images and image filtering CS4670: Computer Vision Kavita Bala Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htmhttp://cvcl.mit.edu/hybridimage.htm

4. CS4670: Computer Vision Kavita Bala Hybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htmhttp://cvcl.mit.edu/hybridimage.htm Lecture 1: Images and image filtering

5. Reading and Announcements Szeliski, Chapter 3.1-3.2 For CMS, wait and then contact Randy Hess (rbhess@cs.cornell.edu)

6. What is an image?

7. Digital Camera The Eye Source: A. Efros We’ll focus on these in this class (More on this process later)

8. What is an image? A grid (matrix) of intensity values (common to use one byte per value: 0 = black, 255 = white) = = 255 200255 75 255 7595 75255 96127145175255 127145175 255 127145200 175 95255 127145200 175 9547255 127145 175127 9547255 74127 95 47255 74 255

9. Images as functions An image contains discrete numbers of pixels Pixel value – grayscale/intensity [0,255] – Color RGB [R, G, B], where [0,255] per channel Lab [L, a, b]: Lightness, a and b are color-opponent dimensions HSV [H, S, V]: Hue, saturation, value

10. Images as functions Can think of image as a function, f, from R 2 to R or R M : – Grayscale: f (x,y) gives intensity at position (x,y) f: [a,b] x [c,d]  [0,255] – Color: f (x,y) = [r(x,y), g(x,y), b(x,y)]

11. A digital image is a discrete (sampled, quantized) version of this function What is an image? x y f (x, y)

12. Image transformations As with any function, we can apply operators to an image g (x,y) = f (x,y) + 20 g (x,y) = f (-x,y)

13. Image transformations As with any function, we can apply operators to an image g (x,y) = f (x,y) + 20 g (x,y) = f (-x,y)

14. Filters Filtering – Form a new image whose pixels are a combination of the original pixels Why? – To get useful information from images E.g., extract edges or contours (to understand shape) – To enhance the image E.g., to blur to remove noise E.g., to sharpen to “enhance image” a la CSI

15. Super-resolution Noise reduction

16. Given a camera and a still scene, how can you reduce noise? Take lots of images and average them! How to formulate as filtering? Source: S. Seitz

17. Image filtering Modify the pixels in an image based on some function of a local neighborhood of each pixel 514 171 5310 Local image data 7 Modified image data Some function S Source: L. Zhang

18. Filters: examples 000 010 000 Original (f) Identical image (g) Source: D. Lowe * = Kernel (k)

19. Filters: examples 111 111 111 Blur (with a mean filter) (g) Source: D. Lowe * = Original (f) Kernel (k)

20. Mean filtering 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 111 111 111 *

21. Mean filtering/Moving average

26. Mean filtering 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 111 111 111 * = 0102030 2010 0204060 4020 0306090 6030 0 5080 906030 0 5080 906030 0203050 604020 102030 2010 00000

27. Mean filtering/Moving Average Replace each pixel with an average of its neighborhood Achieves smoothing effect – Removes sharp features 111 111 111

28. Filters: Thresholding

29. Linear filtering One simple version: linear filtering – Replace each pixel by a linear combination (a weighted sum) of its neighbors – Simple, but powerful – Cross-correlation, convolution The prescription for the linear combination is called the “kernel” (or “mask”, “filter”) 0.5 00 10 000 kernel 8 Modified image data Source: L. Zhang Local image data 614 181 5310

30. Filter Properties Linearity – Weighted sum of original pixel values – Use same set of weights at each point – S[f + g] = S[f] + S[g] – S[k f + m g] = k S[f] + m S[g]

31. Linear Systems Is mean filtering/moving average linear? – Yes Is thresholding linear? – No

32. Filter Properties Linearity – Weighted sum of original pixel values – Use same set of weights at each point – S[f + g] = S[f] + S[g] – S[p f + q g] = p S[f] + q S[g] Shift-invariance – If f[m,n]  g[m,n], then f[m-p,n-q]  g[m-p, n-q] – The operator behaves the same everywhere SS

33. Cross-correlation This is called a cross-correlation operation: Let be the image, be the kernel (of size 2k+1 x 2k+1), and be the output image Can think of as a “dot product” between local neighborhood and kernel for each pixel

34. Convolution Same as cross-correlation, except that the kernel is “flipped” (horizontally and vertically) Convolution is commutative and associative This is called a convolution operation:

35. Cross-correlation This is called a cross-correlation operation: Let be the image, be the kernel (of size 2k+1 x 2k+1), and be the output image Can think of as a “dot product” between local neighborhood and kernel for each pixel

36. Stereo head Camera on a mobile vehicle

37. Example image pair – parallel cameras

38. Intensity profiles Clear correspondence between intensities, but also noise and ambiguity

39. region A Normalized Cross Correlation region B vector a vector b write regions as vectors a b

40. left image band right image band cross correlation 1 0 0.5 x

41. Convolution Same as cross-correlation, except that the kernel is “flipped” (horizontally and vertically) Convolution is commutative and associative This is called a convolution operation:

42. Convolution Adapted from F. Durand