1. 1 Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval 9-April, 2005 Steven C. H. Hoi *, Michael R. Lyu *, Rong Jin # * Department of Computer Science & Engineering The Chinese University of Hong Kong Shatin, N.T., Hong Kong SAR # Department of Computer Science and Engineering Michigan State University East Lansing, MI 48824, USA The 1st IEEE EMMA Workshop in conjunction with 21st IEEE ICDE, Japan, April, 2005.

2. 2 Outline Introduction Background  Log-based Relevance Feedback Coupled Support Vector Machine  Support Vector Machine  Formulation  Alternating Optimization  A Practical Algorithm Experimental Results Conclusion

3. 3 Introduction Content-based Image Retrieval (CBIR)  An important component in visual information retrieval  QBE: query-by-example based on low-level visual features  Semantic gap: low-level features, high-level concepts QBE

4. 4 Introduction Relevance Feedback (RF)  A powerful tool to attack the semantic gap problem  Interactive mechanism to solicit users ’ feedbacks  Boost the retrieval performance of CBIR greatly  Many existing techniques already … Problems  Regular relevance feedback needs too many rounds of interactions for achieving satisfactory results.

5. 5 Introduction Motivation Can user feedback log be used to improve the regular relevance feedback? Relevance Feedback User Feedback Log ? Problem

6. 6 Background Log-based Relevance Feedback (LRF)  Relevance Matrix: R  RF round / Log session: N l images are marked  Elements: relevant (1), irrelevant (-1), unknown (0) Log Sessions Image samples 1 1 0 1 11 1 1 0

7. 7 Background Learning Problem for LRF  Low-level image content:  User feedback log:  Multi-Modal Learning Problem

8. 8 Coupled Support Vector Machine Motivation  How to attack the learning problem on the two modalities? Low-level Image content: X User relevance feedback log: R  Support Vector Machines: superior classification performance A Straightforward Solution:  Learn an SVM classifier on each modality respectively For image content X, we learn an optimal weighting vector w; For log content R, we learn an optimal weighting vector u;  Combine their results together linearly

9. 9 A Straightforward Solution  For the image content modality: w T x  For the user feedback log modality: u T r Coupled Support Vector Machine

10. 10 Disadvantages of the straightforward solution  Linear combination  Modality Consistence Our better solution: Coupled SVM  Learn the two modalities in a unified formulation  Enforce the prediction on the two types of information to be consistent. Coupled Support Vector Machine

11. 11 Formulation: Coupled SVM Coupled Support Vector Machine

12. 12 Optimization of Coupled SVM  Hard to be solved directly  Alternating Optimization (AO) AO: two-step optimization  Fix Y ’, try to find (u, b_u), and (w, b_w)  Fix (u, b_u) and (w, b_w), try to find Y ’ Coupled Support Vector Machine

13. 13 Alternating Optimization  Fix Y ’, the primal optimization is equivalent to solving the two optimization subproblems: Coupled Support Vector Machine

14. 14 Alternating Optimization (AO)  By introducing non-negative Lagrange multipliers, the above two subproblems can be solved Coupled Support Vector Machine

15. 15 Alternating Optimization (AO)  After solving (u, b_u) and (w, b_w), fixing them, the optimal Y ’ can be found to fit the data as follows: Coupled Support Vector Machine

16. 16 Summary of AO procedure  1) Beginning with a small value of  2) Performing the two-step AO procedure  3) Repeating 2) by increasing until it achieves the setting threshold Comments on the Coupled SVM  Can be a general approach for multi-modal learning problems  Need to investigate the convergence issue of Alternating Optimization  Need to study better methods for solving the optimization problem  Require to take some practical considerations when fitting for specific problems. Coupled Support Vector Machine

17. 17 A Practical Algorithm  Practical considerations Cannot engage all unlabeled samples due to response requirement for relevance feedback Strategy for choosing unlabeled samples –Closest to the decision boundary of SVM: most informative according to active learning –Closest to the labeled samples: to avoid too much effort in learning the label information Introducing a parameter to control the error for label correction to avoid overlarge change in the labeled set Coupled Support Vector Machine

18. 18 A Practical Algorithm (cont ’ d) Coupled Support Vector Machine

19. 19 A Practical Algorithm (cont ’ d) Coupled Support Vector Machine

20. 20 Experimental Results Dataset  Images selected from COREL image CDs  Two ground-truth datasets 20-Category: each category contains 100 images, totally 2,000 50-Category: each category contains 100 images, totally 5,000

21. 21 Experimental Results (cont ’ d) Low-level Image Representation  Color Moment 9-dimension  Edge Direction Histogram 18-dimension Canny detector, 18 bins of 20 degrees each  Wavelet-based texture 9-dimension Daubechies-4 wavelet, 3-level DWT Entropies of 9 subimages are generated for the texture feature

22. 22 Experimental Results (cont ’ d) Collection of User Log Data  Log format A log session (LS) corresponds a relevance feedback round Each log session contains 20 images labeled by users  Log data On 20-Category: 161 log sessions On 50-Category: 184 log sessions

23. 23 Experimental Results (cont ’ d) CBIR GUI for collecting feedback data

24. 24 Experimental Results (cont ’ d) Performance Evaluation  Measurement Metric Average Precision = # relevant images / # returned images  Experimental Setting 100 queries 20 initially labeled images SVM: RBF kernel, parameters set via training data  Comparison Schemes RF-SVM –traditional relevance feedback by SVM LRF-2SVM –log-based relevance feedback by learning two SVMs respectively LRF-CSVM –log-based relevance feedback by Coupled SVM

25. 25 Experimental Results (cont ’ d) Performance Evaluation: on 20-Category Dataset

26. 26 Experimental Results (cont ’ d) Performance Evaluation: on 50-Category Dataset

27. 27 Experimental Results (cont ’ d)

28. 28 Experimental Results (cont ’ d)

29. 29 Conclusion A log-based relevance feedback scheme was studied by integrating user feedback log into the content learning of low-level visual features in content-based image retrieval. A general multimodal learning technique, i.e. Coupled Support Vector Machine, was proposed for studying the data with multiple modalities. A practical algorithm by Coupled SVM was presented to attack the log-based relevance feedback problem in CBIR. Experimental results show our proposed scheme is effective for the log-based relevance feedback problem.

30. 30 Q&A

31. 31 References Chu-Hong Hoi and Michael R. Lyu, A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval, in Proc. ACM Multimedia, New York, USA, 10-16 October, pp. 24-31, 2004 S. Tong and E. Chang. Support vector machine active learning for image retrieval. In Proc. ACM Multimedia, pages 107--118, 2001.