1. University of New Orleans 1 Intelligent Indexing and Retrieval of Images A Machine Learning Approach Yixin Chen Dept. of Computer Science and Engineering The Pennsylvania State University http://www.cse.psu.edu/~yixchen

2. University of New Orleans2 Outline Introduction A region-based image categorization method Cluster-based retrieval of images by unsupervised learning Conclusions and future work

3. University of New Orleans3 Image Retrieval The driving forces Internet Storage devices Computing power Two approaches Text-based approach Content-based approach

4. University of New Orleans4 Text-Based Approach Input keywords descriptions Text-Based Image Retrieval System Elephants Image Database

5. University of New Orleans5 Text-Based Approach Index images using keywords (Google, Lycos, etc.) Easy to implement Fast retrieval Web image search (surrounding text) Manual annotation is not always available A picture is worth a thousand words Surrounding text may not describe the image

6. University of New Orleans6 Content-Based Approach Index images using low-level features Content-based image retrieval (CBIR): search pictures as pictures CBIR System Image Database

7. University of New Orleans7 CBIR Applications Commerce (fashion catalogue, ……) Biomedicine (X-ray, CT, ……) Crime prevention (security filtering, ……) Cultural (art galleries, museums, ……) Military (radar, aerial, ……) Entertainment (personal album, ……)

8. University of New Orleans8 CBIR A highly interdisciplinary research area Databases CBIR Information Retrieval Computer Vision Domain Knowledge

9. University of New Orleans9 Previous Work on CBIR Starting from early 1990s General-purpose image search engines IBM QBIC System and MIT Photobook System (two of the earliest systems) VIRAGE System, Columbia VisualSEEK and WebSEEK Systems, UCSB NeTra System, UIUC MARS System, Stanford SIMPLIcity System, NECI PicHunter System, Berkeley Blobworld System, etc.

10. University of New Orleans10 A Data-Flow Diagram Image Database Feature Extraction Compute Similarity Measure Visualization Histogram, color layout, sub-images, regions, etc. Euclidean distance, intersection, shape comparison, region matching, etc. Linear ordering, Projection to 2-D, etc.

11. University of New Orleans11 Open Problem Nature of digital images: arrays of numbers Descriptions of images: high-level concepts Sunset, mountain, dogs, …… Semantic gap Discrepancy between low-level features and high- level concepts High feature similarity may not always correspond to semantic similarity

12. University of New Orleans12 Narrowing the Semantic Gap Imagery features and similarity measure Select effective imagery features [Tieu et al., IEEE CVPR’00] Subjective experiments [Mojsilovic et al., IEEE Trans. IP 9(1)] TigersCars Feature Space

13. University of New Orleans13 Narrowing the Semantic Gap Image Categorization Vacation images [Vailaya et al., IEEE Trans. IP 10(1)] SIMPLIcity [Wang et al., IEEE Trans. PAMI 23(9)] Indoor/ourdoor [Yu and Wolf, SPIE 1995] ALIP [Li et al., IEEE Trans. PAMI 2003] Image Database Indoor Landscape Sunset Outdoor ForestMountain City

14. University of New Orleans14 Narrowing the Semantic Gap Relevance feedback Adjusting similarity measure [Picard et al., IEEE ICIP’96], [Rui et al., IEEE CSVT 8(5)], [Cox et al., IEEE Trans. IP 9(1)] Support vector machine [Tong et al. ACM MM’01] CBIR System 1 2 3 4 1, 2, 3, 4

15. University of New Orleans15 Outline Introduction A region-based image categorization method Cluster-based retrieval of images by unsupervised learning Conclusions and future work

16. University of New Orleans16 Image Categorization Image categorization Labeling of images into one of a number of predefined categories Difficulties Variable and uncontrolled imaging conditions Complex and hard-to-describe objects Occlusion Semantic gap

17. University of New Orleans17 Motivation (a) to (d) belong to winter category since we see snow in them (b) to (f) belong to people category since there are people in them (b) to (d) belong to skiing category since we see people and snow (a) to (g) belong to outdoor scene category since they all have a region or regions corresponding to snow, sky, sea, trees, or grass

18. University of New Orleans18 Our Approach Region-based method Rule-based description Generalization of supervised learning Labels are associated with images instead of individual regions Identical to multiple-instance learning (MIL) setting Images – bags Regions – instances

19. University of New Orleans19 Multiple-Instance Learning Every instances posses an unknown true label, which is only indirectly accessible through bag labels Key assumptions of previous formulation [Andrews, et al., NIPS’03], [Maron, et al., ICML’98], [Zhang, et al., ICML’02] Bags and instances share the same set of labels A bag receives a particular label if at least one of the instances in the bag possesses the label

20. University of New Orleans20 Multiple-Instance Learning Previous formulation does not perform well for image categorization SnowPeopleSkyTrees (a)(e) (f)(e) (f)(a) (f) (g)

21. University of New Orleans21 New Formulation Weaker assumptions Bags and instances DO NOT share the same set of labels {winter, people, skiing, outdoor scenes} {snow, people, sky, sea, trees, grass} Each instance has multiple labels with different weights

22. University of New Orleans22 Learning Region Prototypes (RP) Each RP corresponds to an instance class Each RP is a local maximizer of Diverse Density [Maron et al., NIPS’98] Distance between an RP and an image Rule-based classifier Support vector learning [Chen et al., IEEE Trans. Fuzzy Systems 2003]

23. University of New Orleans23 Experiments 20 thematically diverse image categories, each containing 100 images

24. University of New Orleans24 Categorization Performance Classification accuracy Our approach Color Histogram SVM MIL (old formulation) [Andrews et al., NIPS’03]

25. University of New Orleans25 Categorization Performance Confusion matrix

26. University of New Orleans26 Categorization Performance Some misclassified images

27. University of New Orleans27 Sensitivity to Image Segmentation 6.8% 9.5% 11.7% 13.8% 27.4% Compare our method with MI-SVM

28. University of New Orleans28 Scalability Compare our method with MI-SVM

29. University of New Orleans29 Scalability Difference in classification accuracy between our method and MI-SVM

30. University of New Orleans30 Outline Introduction A region-based image categorization method Cluster-based retrieval of images by unsupervised learning Conclusions and future work

31. University of New Orleans31 Motivation A query image and its top 29 matches returned by a CBIR system Horses (11 out of 29), flowers (7 out of 29), golf player (4 out of 29)

32. University of New Orleans32 CLUE: CLUsters-based rEtrieval of images by unsupervised learning Hypothesis In the “vicinity” of a query image, images tend to be semantically clustered CLUE attempts to capture high-level semantic concepts by learning the way that images of the same semantics are similar

33. University of New Orleans33 System Overview A general diagram of a CBIR system using the CLUE Image Database Feature Extraction Compute Similarity Measure Display And Feedback Select Neighboring Images Image Clustering CLUE

34. University of New Orleans34 Neighboring Images Selection Nearest neighbors method Pick k nearest neighbors of the query as seeds Find r nearest neighbors for each seed Take all distinct images as neighboring images k=3, r=4

35. University of New Orleans35 Weighted Graph Representation Geometric representation Graph representation Vertices denote images Edges are formed between vertices Nonnegative weight of an edge indicates the similarity between two vertices

36. University of New Orleans36 Clustering Graph partitioning and cut Normalized cut (Ncut) [Shi et al., IEEE Trans. PAMI 22(8)]

37. University of New Orleans37 An Experimental System Similarity measure UFM [Chen et al. IEEE PAMI 24(9)] Database COREL 60,000

38. University of New Orleans38 User Interface

39. University of New Orleans39 Query Examples Query Examples from 60,000-image COREL Database Bird, car, food, historical buildings, and soccer game CLUE UFM Bird, 6 out of 11Bird, 3 out of 11

40. University of New Orleans40 Query Examples CLUEUFM Car, 8 out of 11Car, 4 out of 11 Food, 8 out of 11Food, 4 out of 11

41. University of New Orleans41 Query Examples CLUEUFM Historical buildings, 10 out of 11Historical buildings, 8 out of 11 Soccer game, 10 out of 11Soccer game, 4 out of 11

42. University of New Orleans42 Clustering WWW Images Google Image Search Keywords: tiger, Beijing Top 200 returns 4 largest clusters Top 18 images within each cluster

43. University of New Orleans43 Clustering WWW Images Tiger Cluster 1 (75 images) Tiger Cluster 3 (32 images) Tiger Cluster 2 (64 images) Tiger Cluster 4 (24 images)

44. University of New Orleans44 Clustering WWW Images Beijing Cluster 1 (61 images)Beijing Cluster 2 (59 images) Beijing Cluster 3 (43 images)Beijing Cluster 4 (31 images)

45. University of New Orleans45 Retrieval Accuracy 10 image categories each containing 100 images

46. University of New Orleans46 Outline Introduction A region-based image categorization method Cluster-based retrieval of images by unsupervised learning Conclusions and future work

47. University of New Orleans47 Conclusions Two approaches to tackle the “semantic gap” problem Region-based image categorization method Retrieving image clusters by unsupervised learning Tested using 60,000 images from COREL and images from WWW

48. University of New Orleans48 Potential Applications Biomedical Informatics MRI, CT images Internet Image search engine Digital libraries

49. University of New Orleans49 Future Work Continue to make contributions to the areas of Machine learning theories Computer vision Robotics Computational intelligence

50. University of New Orleans50 Future Work Apply theories to real world problems Biomedical Informatics GIS Robot vision systems Sensor, imaging and video systems Autonomous intelligent agents

51. University of New Orleans51 Related Publications Image Categorization IEEE Trans. Fuzzy Systems, 11, 2003 IEEE Int’l Conf. on Fuzzy Systems, 2003 Machine Learning Research, (ready for submission) CLUE IEEE Int’l Sym. Signal Processing and its Applications, 2003 ACM Multimedia Conference, 2003 (under review) IEEE Trans. Pattern Anal. Machine Intell., (under review) Unified Feature Matching IEEE Trans. Pattern Anal. Machine Intell., 24(9),2003 ACM Multimedia Conference, 2001 IEEE Int’l Conf. on Fuzzy Systems, 2003

52. University of New Orleans52 Support by The National Science Foundation The Pennsylvania State University The PNC Foundation SUN Microsystems NEC Research Institute

53. University of New Orleans53 Acknowledgment Dissertation committee at Penn State Professor James Z. Wang, IST and CSE Professor Lee C. Giles, IST and CSE Professor John Yen, IST and CSE Professor Jia Li, STAT Professor Donald Richards, STAT NEC Research Institute Dr. Robert Krovetz

54. University of New Orleans54 More Information Papers in PDF, 60,000-image DB, demonstrations, etc. http://wang.ist.psu.edu http://www.cse.psu.edu/~yixchen