1 Overview of Image Retrieval Hui-Ying Wang

2/42 Reference Smeulders, A. W., Worring, M., Santini, S., Gupta, A.,, and Jain, R “Content-based image retrieval at the end of the early years.” IEEE Trans. Pattern Analysis and Machine Intelligence 22, 12, 1349–1380. R. Datta, D. Joshi, J. Li and J. Z. Wang, ”Image Retrieval: Ideas, Influences, and Trends of the New Age,” ACM Computing Surveys, 2008, to appear. CVPR 2007 short course: Recognizing and Learning Object Categories x.html

3/42 Outline Motive Academic and real world Difficulties and problem model Evaluation metrics

4/42 Outline Motive Academic and real world Difficulties and problem model Evaluation metrics

5/42 Motive Popular electronic device –Digital camera By-product –Digital photos Need –Organization Key: filenames? dates?

6/42 Outline Motive Academic and real world Difficulties and problem model Evaluation metrics

7/42 Publications in CBIR

8/42 Real-world system Search engines –Google Image (2.2 b) –Picsearch (1.7 b) –Yahoo! Images (1.6 b) –AltaVista –Ask Images Online albums –Flickr –Riya –Webshots Shopping –like

9/42 Real-world system Search engines –Google Image (2.2 b) –Picsearch (1.7 b) –Yahoo! Images (1.6 b) –AltaVista –Ask Images Online albums –Flickr –Riya –Webshots Shopping –like

10/42 Google Images

11/42 Google Image Labeler

12/42 Picsearch

13/42 Yahoo! Images

14/42 AltaVista

15/42 Ask Images

16/42 Real-world system Search engines –Google Image (2.2 b) –Picsearch (1.7 b) –Yahoo! Images (1.6 b) –AltaVista –Ask Images Online albums –Flickr –Riya –Webshots Shopping –like

17/42 Flickr

18/42 Webshots

19/42 Riya

20/42 Real-world system Search engines –Google Image (2.2 b) –Picsearch (1.7 b) –Yahoo! Images (1.6 b) –AltaVista –Ask Images Online albums –Flickr –Riya –Webshots Shopping –like

21/42 like

22/42 Outline Motive Academic and real world Difficulties and problem model Evaluation metrics

23/42 Challenges view point variation scale illumination deformation occlusion

24/42 Goal real object sensory gap digital record interpretation semantic gap extraction human vision computer vision

25/42 Core problems How to describe an image How to assess the similarity

26/42 Some features Global features –MPEG-7 Color Layout Descriptor Edge Histogram Descriptor Homogeneous Texture Descriptor Summarizing local features –Bag of Features

27/42 Some features Global features –MPEG-7 Color Layout Descriptor Edge Histogram Descriptor Homogeneous Texture Descriptor Summarizing local features –Bag of Features

28/42 Color Layout Descriptor - Presentation MPEG-7

29/42 Color Layout Descriptor - Similarity

30/42 Some features Global features –MPEG-7 Color Layout Descriptor Edge Histogram Descriptor Homogeneous Texture Descriptor Summarizing local features –Bag of Features

31/42 Edge Histogram Descriptor - Presentation MPEG-7

32/42 Edge Histogram Descriptor - Similarity

33/42 Some features Global features –MPEG-7 Color Layout Descriptor Edge Histogram Descriptor Homogeneous Texture Descriptor Summarizing local features –Bag of Features

34/42 Homogeneous Texture Descriptor - Presentation e: log-scaled sum of the squares of Gabor-filtered Fourier transform coefficients d: log-scaled standard deviation of the squares of Gabor-filtered Fourier transform coefficients Human Vision System Fourier transform Gabor function f DC : mean deviation f SD : standard deviation

35/42 Homogeneous Texture Descriptor - Similarity

36/42 Some features Global features –MPEG-7 Color Layout Descriptor Edge Histogram Descriptor Homogeneous Texture Descriptor Summarizing local features –Bag of Features

37/42 Local feature Detected keypoints –spatial relationship fully independent (ex: bag of features) fully connected

38/42 Bag of Features

39/42 Outline Motive Academic and real world Difficulties and problem model Evaluation metrics

40/42 Evaluation (1/2) Standard –Precision # of retrieved positive images / # of total retrieved images –Recall # of retrieved positive images / # of total positive images

41/42 Evaluation (1/2) When number of retrieved images increase –Recall ↑ Precision ↓ Average precision (AP) –The area under the precision-recall curve for a query precision recall 1 1 AP

42 The end ~ Thank you