1. Personalized Social Image Recommendation
Jianping Fan
Department of Computer Science
University of North Carolina at Charlotte

2. Social Image Sharing, Tagging, Friendship, ….

3. Social Image Sharing, Tagging, Friendship, ….

4. What people may care? Entertainment Tagging Interesting Photos
Making friends by tagging & sharing
Knowing hot topics
Navigating relevant photos ……

5. Significant Difference with News
Social Images & Tags
Information are provided by non-professionals
Information filtering and noise reductions are necessary
Personalized service is more important

6. 1. Motivation Large amount of images are available on Internet
Flickr.com is becoming very popular
Hard for Query Formulation
b. Information Overload

7. 1. Motivation Applications Second-Generation Internet Search Engines
Family Photo Organization System
Digital Library

8. 2. Key Issues to be Addressed
How can we provide a good global overview of large-scale image collections such as Flickr images?
How can we capture the users’ query intentions easily and effectively?
How can we recommend the most relevant images according to the users’ intentions?

9. Why Flickr is attractive?
Low barrier to enter, low cognitive cost
Immediate feedback and comments
Individual & community aspects

10. What problems Flickr have?
Low Quality of Tagging: people may make mistakes
Homonymy problem: one text term may have different means under different contexts
Synonymy problem: many text terms may have same mean
Tag redundancy

11. 2. Key Issues to be Addressed
Query Recommendation: picture owners may use different text terms for image annotation, and users may not know these text terms
Information overload: each text term may relate to many images in Flickr such as Roma

12. What Flickr has done for these?
Tagcloud for query recommendation

13. Problems for Tagcloud
Only the frequency of text terms is used to characterized their importance, but the importance of the text terms may also depend on other issues
Contexts between the text terms are missed, thus it cannot support effective navigation and exploration
Such tagcloud may not be able to represent the users’ image needs effectively

14. Some improvements on tag vis.

15. Some improvements on tag vis.

16. Some improvements on tag vis.

17. Social Network Visualization
How to identify such contexts for social tags?

18. Our Proposed Solutions
Topic Network for providing a good global overview of large-scale image collections;
Interactive Visualization for capturing the users’ query intentions easily and effectively;
Interestingness Matching for recommending the most relevant images according to the users’ intentions.

19. 3. Topic Network Construction
What can topic network represent?
Global overview of large-scale image collections;
Context among the topics; local contexts + global contexts
Good navigation or browsing structure for accessing large-scale image collections.

20. 3. Topic Network Construction
Major steps for concept network construction
Natural language processing for entity extraction;
Word frequency estimation;
Keyword extraction;
Co-occurrence for context determination.

21. 3. Topic Network Construction
Social Tags & Discussions in Texts
Social Tag Cleansing
Topics & Topic Correlations
Topic Network

22. 3. Topic Network Construction
Social Tags & Discussions in Texts
Social Tag Cleansing
Keyword Extraction
Correlation Analysis
Topics & Topic Correlations

23. 3. Topic Network Construction
Social Tags & Discussions in Texts
Social Tag Cleansing
Spam tags
Noisy tags
………..
Topics & Topic Correlations

24. 3. Topic Network Construction
Social Tags & Discussions in Texts
Keyword Extraction
Numbers of images
Numbers of discussions
Numbers of clicks
……….
Topics & Topic Correlations

25. 3. Topic Network Construction
Co-Occurrence Probability
Semantic Similarity
Distribution Similarity

26. 3. Topic Network Construction
Context between concepts

27. 3. Topic Network Construction

28. 3. Topic Network Construction

29. 4. Topic Network Visualization
Interestingness of topic
self-significance: numbers of images, discussions & clicks
Link-significance: numbers of linkages

30. 4. Topic Network Visualization
Using interestingness to highlight image topics

31. 4. Topic Network Visualization
Constraint-based topic clustering

32. 4. Topic Network Visualization
Constraint-based topic clustering

33. 4. Topic Network Visualization
Query-Driven Topic Network Visualization

34. 4. Topic Network Visualization
Query-Driven Topic Network Visualization

35. 4. Topic Network Visualization
Multidimensional Scaling

36. 4. Topic Network Visualization

37. 4. Topic Network Visualization
Change of Focus

38. 4. Topic Network Visualization
Change of Focus---Poincare Mapping

39. 4. Topic Network Visualization
Change of Focus---Poincare Mapping

40. 5. Interactive Query Specification
Concept Network Navigation

41. 5. Interactive Query Specification
Personalized concept network generation

42. 5. Interactive Query Specification

43. 6. Most Representative Image Recommendation
Image Content Representation

44. 6. Most Representative Image Recommendation
Image Similarity Characterization
Color Histogram Kernel

45. 6. Most Representative Image Recommendation
Image Similarity Characterization
Wavelet Filter Bank Kernel

46. 6. Most Representative Image Recommendation
Image Similarity Characterization
Interest Point Matching Kernel

47. 6. Most Representative Image Recommendation
Image Similarity Characterization
---- mixture of kernels
Multiple kernel learning for determining weighting parameters!

48. 6. Most Representative Image Recommendation
Kernel-Based Image Clustering

49. 6. Most Representative Image Recommendation
Support Vector Machine
positive images
negative images
positive support vectors
margin
negative support vectors

50. 6. Most Representative Image Recommendation
Representativeness of image

51. 6. Most Representative Image Recommendation
Most Representative Image Selection
Most representative images come from all these clusters and distribute according their coverage percentages;
Images close to decision boundaries;
Most frequent accessed images

52. 6. Most Representative Image Visualization
Kernel PCA
Similarity-Preserving Image Projection

53. 6. Most Representative Image Visualization
Hyperbolic Image Visualization

54. 6. Most Representative Image Visualization

55. 6. Most Representative Image Visualization

56. 6. Most Representative Image Visualization

57. 7. User-Adaptive Image Recommendation
User Intention Capturing

58. 7. User-Adaptive Image Recommendation
Personalized Interestingness of Image

59. 7. User-Adaptive Image Recommendation

60. 7. User-Adaptive Image Recommendation

61. Collaborative Filtering
Social Rating Information
User’s Personal History
Potential Changes over time
Social Network

62. Multi-Modal Deep Learning