1. Jianping Fan Department of Computer Science University of North Carolina at Charlotte Charlotte, NC 28223 Relevance Feedback for Image Retrieval

2. 1. How to Build Image Database? a. Taking whole frame as an object b. Extracting objects or regions from images

3. 1. How to Build Image Database? c. Feature Extraction Color Texture Shape

4. 1. How to Build Image Database? d. Image Clustering

5. Images in Database Cluster 1 Cluster i Cluster n Subcluster i1Subcluster ijSubcluster im Subregion ij1Subregion ijlSubregion ijp images 1. How to Build Image Database? e. Database Indexing

6. Query By Examples Query exampleRanked Results

8. Query-By-Example Query Example Feature Extraction Within-Node Nearest Neighbor Search Distance Function Top-K Results

9. 2. What’s Relevance Feedback? a.The client send his/her request to the database system; b.The database system sends him/her some ranked answers; c.The client can exchange his/her judgment with the system. no

10. 2. What’s Relevance Feedback?

12. 3. Relevance Feedback Distance Weighting Approach: Query Example Feature Extraction Database Indexing

13. Query Example Feature Extraction Within-Node Nearest Neighbor Search Distance Function Top-K Results 3. Relevance Feedback

14. Effectiveness of Feature Weighting: Original Feature Space Weighted Feature Space

15. 3. Relevance Feedback Two More Issues for Feature Weighting a.Informative Sample Generation: what we should return to users, so that they can make good decision on relevance vs. irrelevance? b.Query Movement Control: Through weighting the features, it is able for us to control the importance between the features for image similarity characterization. However, for image retrieval application, we also need to control the query point to move to target in the best way!

16. 3. Relevance Feedback

17. Query Updating New Query Vector Previous Query Vector Vectors for Positive Images Vectors for Negatives

18. 3. Relevance Feedback Query Point Movement Control Initial Query Point Target Image Best Search Road Potential Convergence Search Road Where to go? No Convergence

19. Informative Image Sampling ? 3. Relevance Feedback

20. 4. MEGA System in UCSB a. Initialize the query

21. 4. MEGA System in UCSB b. Send the query to system

22. 4. MEGA System in UCSB c. Client mark the relevant examples

23. 4. MEGA System in UCSB d. System Evaluation according to client feedback

24. 4. MEGA System in UCSB e. Second client feedback

25. 4. MEGA System in UCSB f. Second System Evaluation

26. Problem for Feature Weighting Approach a.Cost-Sensitive: It is very expensive to update the feature weights on real time! b. Semantic Gap: The distance functions may not be able to characterize the underlying image similarity effectively! c. Visualization: The underlying image display tools may separate similar images in different places, it is hard for users to evaluate the visual similarity (relevance) between the images! 3. Relevance Feedback

27. Challenging Issues d. Convergence: It is very important to guarantee the algorithm for kernel updating is converged! e. Cost Reduction: It is very important to reduce the cost for kernel updating! 3. Relevance Feedback

28. Query is initialized by keyword Kernel-Based Clustering of Google Search Results Similarity-Based Image Projection and Visualization Intention capturing and Kernel Selection for Junk Image Filtering Relevance is user-dependent! 4. Relevance Feedback for Query by Keywords

29. Requirements for such new search engine: Fast algorithm for feature extraction; Multiple kernels for diverse image similarity characterization; Implicit query intention capturing and real-time kernel updating 4. Relevance Feedback for Query by Keywords

30. Keyword-Based Google Images Search Fast Feature Extraction & Basic Kernels Mixture-of-Kernels & Image Clustering Hyperbolic Image Visualization Query Intention Expression & Hypothesis Making Accept? No Increment Kernel Learning Through incremental learning, we can consider multiple competing hypotheses for the same task! 4. Relevance Feedback for Query by Keywords

31. Fast Feature Extraction 4. Relevance Feedback for Query by Keywords

32. Image Representation & Similarity a. Color histogram for whole image b. 10 color histograms for different patterns c. Wavelet transformation 4. Relevance Feedback for Query by Keywords

33. points in HD Space They are invisible for human eye! images 4. Relevance Feedback for Query by Keywords

34. Basic kernels for image similarity characterization: Color Histogram Kernel Wavelet Filter Bank Kernel Sub-Image Color Histogram Kernel 4. Relevance Feedback for Query by Keywords

35. Mixture-of-kernels for diverse similarity characterization: (a) It could be expensive for learning a good combination! (b) The similarity between the images depends on the given kernel function! 4. Relevance Feedback for Query by Keywords

36. Hypothesis Making & Initial Analysis subject to: Decision function: R Outliers Majority 4. Relevance Feedback for Query by Keywords

37. Similarity-Preserving Image Projection Transform large amount of images (represented by high-dimensional visual features) into their similarity contexts for enabling better visualization! 4. Relevance Feedback for Query by Keywords

38. Hyperbolic Image Visualization & Hypothesis Assessment projection Invisible HD SpaceVisible 2D Disk Unit 4. Relevance Feedback for Query by Keywords

39. Mountain 4. Relevance Feedback for Query by Keywords

40. Ocean 4. Relevance Feedback for Query by Keywords

41. Sunrise 4. Relevance Feedback for Query by Keywords

42. Grass 4. Relevance Feedback for Query by Keywords

43. User-System Interaction for Making New Hypothesis Hypothesis-Driven Image Re-Clustering 4. Relevance Feedback for Query by Keywords

44. Hypothesis-Driven Data Analysis: a. Updating decision function: margin between relevant images and irrelevant images! b. Updating the combination of feature subsets! c. Updating image projection optimization criteria to obtain more accurate projection! d. Updating image representation! 4. Relevance Feedback for Query by Keywords

45. Incremental Learning: Update decision function Dual Problem Subject to: 4. Relevance Feedback for Query by Keywords

46. Incremental Learning: Update decision function a. Old decision function b. New decision function with user’s feedbacks 4. Relevance Feedback for Query by Keywords

47. Incremental Learning: Update Feature Weights 4. Relevance Feedback for Query by Keywords

48. Make the decision function to be visible! 4. Relevance Feedback for Query by Keywords

49. Enlarge the margin between two classes! 4. Relevance Feedback for Query by Keywords

50. Enlarge the margin between two classes! 4. Relevance Feedback for Query by Keywords

51. Larger margin has good generalization property! 4. Relevance Feedback for Query by Keywords

52. Red Rose Forest 4. Relevance Feedback for Query by Keywords

53. Red Flower Sailing 4. Relevance Feedback for Query by Keywords

54. Convergence for Incremental Learning 2000 queries over Google Images Control & reduce users’ efforts! 4. Relevance Feedback for Query by Keywords

55. Incremental Learning is critical for Visual Analytics 4. Relevance Feedback for Query by Keywords

56. Future Work for Relevance Feedback