1. Advanced Multimedia Image Content Analysis Tamara Berg

2. Announcements Start thinking about project ideas – These can be related to text, sound, images, combinations of different media, interaction with digital media, social media, … – Next week on March 10 or 12 visit office hours to discuss your project ideas (20 points for coming with a well thought out idea or ideas). – Can work alone or in pairs. – Project proposal presentations March 17 Reminder – Assignment due March 12 (a week from today).

3. Possible Project Ideas Build a spam/ham detector Extend your pagerank algorithm to do web search by query Build a document topic classification system Implement a music retrieval system based on low level audio features or pitch class profile Implement an image retrieval system Projects related to combined sources of information – images + maps, images + text Something cool with Twitter? Example – twittering the superbowl, or twitter + location information/maps. Projects related to interactivity Build a facebook app to do …

4. How are images stored?

5. Reminder: Images Images are sampled and quantized measurements of light hitting a sensor. What do we mean by sampled? What is being quantized?

6. Images in the computer

8. Color Images Ellsworth Kelly Red Blue Green, 1963 img =

9. Color Images Ellsworth Kelly Red Blue Green, 1963 N img =

10. Color Images Ellsworth Kelly Red Blue Green, 1963 N M img =

11. Color Images Stored as 3d matrix of r, g, and b values at each pixel – Image matrix would be size NxMx3 – R = img(:,:,1)G = img(:,:,2)B = img(:,:,3) Ellsworth Kelly Red Blue Green, 1963 N M img =

12. Red component img(:,:,1) =

13. Green component img(:,:,2) =

14. Blue component img(:,:,3) =

15. Color Images Stored as 3d matrix of r, g, and b values at each pixel – So img(i,j,:) = [r g b] Ellsworth Kelly Red Blue Green, 1963 N M img =

16. Color Images Stored as 3d matrix of r, g, and b values at each pixel – So img(i,j,:) = [r g b]. – In the case above this might be [255 0 0]. Ellsworth Kelly Red Blue Green, 1963 N M img = i j

17. Useful Matlab image functions img = imread(filename); – read in an image imagesc(img); – display an image imwrite(img,outfilename); – write an image img(i,j,:) indexes into the ith row, jth column of the image. subimg = img(1:10,20:30,:) extracts part of img.

18. Matlab demo 1 This will be very useful for homework 3!

19. How should we represent them?

20. Motivation Image retrieval – We have a database of images – We have a query image – We want to find those images in our database that are most similar to the query

21. Motivation Image retrieval – We have a database of images – We have a query image – We want to find those images in our database that are most similar to the query Similarly to text retrieval, & music retrieval we first need a representation for our data.

22. How should we represent an image?

23. First try Just represent the image by all its pixel values

24. First try Img1 = Img2 = Say we measure similarity as: sim = sum(abs(img1 – img2))

25. First try How similar are these two images? Is this bad? Img1 = Img2 = Say we measure similarity as: sim = average diff between values in img1 and img2

26. Matlab demo 2

27. What do we want? Features should be robust to small changes in the image such as: – Translation – Rotation – Illumination changes

28. Second Try Represent the image as its average pixel color Photo by: marielitomarielito

29. Second Try Represent the image as its average pixel color Pros? Cons? Photo by: marielitomarielito

30. Third Try Represent the image as a spatial grid of average pixel colors Pros? Cons? Photo by: marielitomarielito

31. QBIC system First content based image retrieval system – Query by image content (QBIC) – IBM 1995 – QBIC interprets the virtual canvas as a grid of coloured areas, then matches this grid to other images stored in the database. QBIC link

32. Matlab demo 3

33. Color is not always enough! The representation of these two umbrella’s should be similar…. Under a color based representation they look completely different!

34. What next? Edges! But what are they & how do we find them?

35. Reminder: Convolution

36. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 f = g = signal filter

37. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.25

38. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.253

39. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.2533.75

40. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.2533.753.25

41. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.2533.753.252.75

42. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.2533.753.252.753.75- Convolution computes a weighted average.

43. Images -> 2d filtering

44. Let’s replace each pixel with a weighted average of its neighborhood The weights are called the filter kernel What are the weights for a 3x3 moving average? Moving average Source: D. Lowe

45. Let’s replace each pixel with a weighted average of its neighborhood The weights are called the filter kernel What are the weights for a 3x3 moving average? Moving average 111 111 111 “box filter” Source: D. Lowe

46. Defining convolution in 2d f Let f be the image and g be the kernel. The output of convolving f with g is denoted f * g. Source: F. Durand Convention: kernel is “flipped” MATLAB: conv2 vs. filter2 (also imfilter)

47. 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 111 111 111 “box filter” g(x,y)

48. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0 0000000000 0000000000 000 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

49. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 010 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

50. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 01020 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

51. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0102030 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

52. Moving Average In 2D 0102030 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 Source: S. Seitz

53. Moving Average In 2D 0000000000 0000000000 00090 00 000 00 000 00 000 0 00 000 00 0000000000 00 0000000 0000000000 0102030 2010 0204060 4020 0306090 6030 0 5080 906030 0 5080 906030 0203050 604020 102030 2010 00000 Source: S. Seitz What is this filter doing?

54. Practice with linear filters 000 010 000 Original ? Source: D. Lowe

55. Practice with linear filters 000 010 000 Original Filtered (no change) Source: D. Lowe

56. Practice with linear filters 000 100 000 Original ? Source: D. Lowe

57. Practice with linear filters 000 100 000 Original Shifted left By 1 pixel Source: D. Lowe

58. Practice with linear filters Original ? 111 111 111 Source: D. Lowe

59. Practice with linear filters Original 111 111 111 Blur (with a box filter) Source: D. Lowe

60. Gaussian Kernel Constant factor at front makes volume sum to 1 (can be ignored, as we should re-normalize weights to sum to 1 in any case) 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003 5 x 5,  = 1 Source: C. Rasmussen

61. Example: Smoothing with a Gaussian source: Svetlana Lazebnik

62. Edges

63. Edge detection Goal: Identify sudden changes (discontinuities) in an image Intuitively, most semantic and shape information from the image can be encoded in the edges More compact than pixels Ideal: artist’s line drawing (but artist is also using object-level knowledge) Source: D. Lowe

64. Origin of Edges Edges are caused by a variety of factors depth discontinuity surface color discontinuity illumination discontinuity surface normal discontinuity Source: Steve Seitz

65. Characterizing edges An edge is a place of rapid change in the image intensity function image intensity function (along horizontal scanline) first derivative edges correspond to extrema of derivative source: Svetlana Lazebnik

66. Edge filters Approximations of derivative filters: Source: K. Grauman Convolve filter with image to get edge map

67. Edge filters Approximations of derivative filters: Source: K. Grauman Respond highly to vertical edges

68. Edge filters Approximations of derivative filters: Source: K. Grauman Respond highly to horizontal edges

69. Edges: example source: Svetlana Lazebnik

70. What about our umbrellas? The representation of these two umbrella’s should be similar…. Under a color based representation they look completely different! How about using edges?

71. Edges Edges extracted using convolution with Prewitt filter Red umbrellaGray umbrella

72. Edges Edges overlayed from red and gray umbrellas. How is this?

73. Edge Energy in Spatial Grid Red UmbrellaGray Umbrella How is this representation?

74. Quick overview of other common kinds of Features

75. Important concept: Histograms Graphical display of tabulated frequencies, shown as bars. It shows what proportion of cases fall into each of several categories. The categories are usually specified as non- overlapping intervals of some variable.

76. Color Histograms Representation of the distribution of colors in an image, derived by counting the number of pixels of each of given set of color ranges in a typically (3D) color space (RGB, HSV etc). What are the bins in this histogram?

77. Shape Descriptors: shape context Representation of the local shape around a feature location (star) – as histogram of edge points in an image relative to that location. Computed by counting the edge points in a log polar space. So what are the bins of this histogram?

78. Descriptor computation: Divide patch into 4x4 sub-patches Compute histogram of gradient orientations (convolve with filters that respond to edges in different directions) inside each subpatch Resulting descriptor: 4x4x8 = 128 dimensions Shape descriptors: SIFT David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004."Distinctive image features from scale-invariant keypoints.” source: Svetlana Lazebnik

79. Texture Features Universal texton dictionary histogram Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003 Convolve with various filters – spot, oriented bar. Compute histogram of responses.