1. Advanced Multimedia Image Content Analysis Tamara Berg

2. Announcements HW2 due March 13, 11:59pm – Questions? HW3 online Thurs

3. How are images stored?

4. Images Images are sampled and quantized measurements of light hitting a sensor.

5. Images in the computer

7. Color Images Ellsworth Kelly Red Blue Green, 1963 img =

8. Color Images Ellsworth Kelly Red Blue Green, 1963 N img =

9. Color Images Ellsworth Kelly Red Blue Green, 1963 N M img =

10. Color Images Stored as 3d matrix of r, g, and b values at each pixel – Image matrix would be size NxMx3 – ? = img(:,:,1)? = img(:,:,2)? = img(:,:,3) Ellsworth Kelly Red Blue Green, 1963 N M img =

11. Color Images Stored as 3d matrix of r, g, and b values at each pixel – Image matrix would be size NxMx3 – R = img(:,:,1)G = img(:,:,2)B = img(:,:,3) Ellsworth Kelly Red Blue Green, 1963 N M img =

12. Red component img(:,:,1) =

13. Green component img(:,:,2) =

14. Blue component img(:,:,3) =

15. Color Images Stored as 3d matrix of r, g, and b values at each pixel – How could you get the r,g,b values for pixel (i,j)? Ellsworth Kelly Red Blue Green, 1963 N M img =

16. Color Images Stored as 3d matrix of r, g, and b values at each pixel vals = img(i,j,:); Then r= vals(1); g = vals(2); b = vals(3); Ellsworth Kelly Red Blue Green, 1963 N M img =

17. Color Images Stored as 3d matrix of r, g, and b values at each pixel – So r,g,b values are stored in img(i,j,:). – In the case above these values might be [1 0 0]. Ellsworth Kelly Red Blue Green, 1963 N M img = i j

18. RGB color space RGB cube – Easy for devices – But not perceptual – Where do the grays live? – Where is hue and saturation? Slide by Steve Seitz

19. HSV Cone Hue, Saturation, Value (Intensity/Brightness ) Use rgb2hsv() and hsv2rgb() in Matlab

20. How should we represent images?

21. Motivation Image retrieval – We have a database of images – We have a query image – We want to find those images in our database that are most similar to the query

22. Motivation Image retrieval – We have a database of images – We have a query image – We want to find those images in our database that are most similar to the query Similarly to text retrieval, & music retrieval we first need a representation for our data.

23. How should we represent an image?

24. First try Just represent the image by all its pixel values

25. First try Img1 = Img2 = Say we measure similarity as: sim = sum(abs(img1 – img2))

26. First try How similar are these two images? Is this bad? Img1 = Img2 = Say we measure similarity as: sim = average diff between values in img1 and img2

27. What do we want? Features should be robust to small changes in the image such as: – Translation – Rotation – Illumination changes

28. Second Try Represent the image as its average pixel color Photo by: marielitomarielito

29. Second Try Represent the image as its average pixel color Pros? Cons? Photo by: marielitomarielito

30. Third Try Represent the image as a spatial grid of average pixel colors Pros? Cons? Photo by: marielitomarielito

31. QBIC system First content based image retrieval system – Query by image content (QBIC) – IBM 1995 – QBIC interprets the virtual canvas as a grid of coloured areas, then matches this grid to other images stored in the database. QBIC link

32. Color is not always enough! The representation of these two umbrella’s should be similar…. Under a color based representation they look completely different!

33. What next? Edges! But what are they & how do we find them?

34. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 f = g = signal filter

35. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.25

36. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.253

37. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.2533.75

38. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.2533.753.25

39. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.2533.753.252.75

40. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.2533.753.252.753.75- Convolution computes a weighted average.

41. Images -> 2d filtering

42. Let’s replace each pixel with a weighted average of its neighborhood The weights are called the filter kernel What are the weights for a 3x3 unweighted moving average? Moving average Source: D. Lowe

43. Let’s replace each pixel with a weighted average of its neighborhood The weights are called the filter kernel What are the weights for a 3x3 unweighted moving average? Moving average 111 111 111 “box filter” Source: D. Lowe

44. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 111 111 111 “box filter” g(x,y)

45. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0 0000000000 0000000000 000 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

46. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 010 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

47. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 01020 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

48. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0102030 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

49. Moving Average In 2D 0102030 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

50. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0102030 2010 0204060 4020 0306090 6030 0 5080 906030 0 5080 906030 0203050 604020 102030 2010 00000 Source: S. Seitz What is this filter doing?

51. Practice with linear filters 000 010 000 Original ? Source: D. Lowe

52. Practice with linear filters 000 010 000 Original Filtered (no change) Source: D. Lowe

53. Practice with linear filters 000 100 000 Original ? Source: D. Lowe

54. Practice with linear filters 000 100 000 Original Shifted left By 1 pixel Source: D. Lowe

55. Practice with linear filters Original ? 111 111 111 Source: D. Lowe

56. Practice with linear filters Original 111 111 111 Blur (with a box filter) Source: D. Lowe

57. Gaussian Kernel Constant factor at front makes volume sum to 1 (can be ignored, as we should re-normalize weights to sum to 1 in any case) 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003 5 x 5,  = 1 Source: C. Rasmussen

58. Example: Smoothing with a Gaussian source: Svetlana Lazebnik

59. Edges

60. Edge detection Goal: Identify sudden changes (discontinuities) in an image Intuitively, most semantic and shape information from the image can be encoded in the edges More compact than pixels Ideal: artist’s line drawing (but artist is also using object-level knowledge) Source: D. Lowe

61. Origin of Edges Edges are caused by a variety of factors depth discontinuity surface color discontinuity illumination discontinuity surface normal discontinuity Source: Steve Seitz

62. Characterizing edges An edge is a place of rapid change in the image intensity function image intensity function (along horizontal scanline) first derivative edges correspond to extrema of derivative source: Svetlana Lazebnik

63. Edge filters Approximations of derivative filters: Source: K. Grauman Convolve filter with image to get edge map

64. Edge filters Approximations of derivative filters: Source: K. Grauman Respond highly to vertical edges

65. Edge filters Approximations of derivative filters: Source: K. Grauman Respond highly to horizontal edges

66. Edges: example source: Svetlana Lazebnik

67. What about our umbrellas? The representation of these two umbrella’s should be similar…. Under a color based representation they look completely different! How about using edges?

68. Edges Edges extracted using convolution with Prewitt filter Red umbrellaGray umbrella

69. Edges Edges overlayed from red and gray umbrellas. How is this?

70. Edge Energy in Spatial Grid Red UmbrellaGray Umbrella How is this representation?

71. Quick overview of other common kinds of Features

72. Important concept: Histograms Graphical display of tabulated frequencies, shown as bars. It shows what proportion of cases fall into each of several categories. The categories are usually specified as non- overlapping intervals of some variable.

73. Color Histograms Representation of the distribution of colors in an image, derived by counting the number of pixels of each of given set of color ranges in a typically (3D) color space (RGB, HSV etc). What are the bins in this histogram?

74. Shape Descriptors: shape context Representation of the local shape around a feature location (star) – as histogram of edge points in an image relative to that location. Computed by counting the edge points in a log polar space. So what are the bins of this histogram?

75. Descriptor computation: Divide patch into 4x4 sub-patches Compute histogram of gradient orientations (convolve with filters that respond to edges in different directions) inside each subpatch Resulting descriptor: 4x4x8 = 128 dimensions Shape descriptors: SIFT David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004."Distinctive image features from scale-invariant keypoints.” source: Svetlana Lazebnik

76. Texture Features Universal texton dictionary histogram Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003 Convolve with various filters – spot, oriented bar. Compute histogram of responses.