Personalized Social Image Recommendation Jianping Fan Department of Computer Science University of North Carolina at Charlotte http://www.cs.uncc.edu/~jfan

Social Image Sharing, Tagging, Friendship, ….

Social Image Sharing, Tagging, Friendship, ….

What people may care? Entertainment Tagging Interesting Photos Making friends by tagging & sharing Knowing hot topics Navigating relevant photos ……

Significant Difference with News Social Images & Tags Information are provided by non-professionals Information filtering and noise reductions are necessary Personalized service is more important

1. Motivation Large amount of images are available on Internet Flickr.com is becoming very popular Hard for Query Formulation b. Information Overload

1. Motivation Applications Second-Generation Internet Search Engines Family Photo Organization System Digital Library

2. Key Issues to be Addressed How can we provide a good global overview of large-scale image collections such as Flickr images? How can we capture the users’ query intentions easily and effectively? How can we recommend the most relevant images according to the users’ intentions?

Why Flickr is attractive? Low barrier to enter, low cognitive cost Immediate feedback and comments Individual & community aspects

What problems Flickr have? Low Quality of Tagging: people may make mistakes Homonymy problem: one text term may have different means under different contexts Synonymy problem: many text terms may have same mean Tag redundancy

2. Key Issues to be Addressed Query Recommendation: picture owners may use different text terms for image annotation, and users may not know these text terms Information overload: each text term may relate to many images in Flickr such as Roma

What Flickr has done for these? Tagcloud for query recommendation

Problems for Tagcloud Only the frequency of text terms is used to characterized their importance, but the importance of the text terms may also depend on other issues Contexts between the text terms are missed, thus it cannot support effective navigation and exploration Such tagcloud may not be able to represent the users’ image needs effectively

Some improvements on tag vis.

Some improvements on tag vis.

Some improvements on tag vis.

Social Network Visualization How to identify such contexts for social tags?

Our Proposed Solutions Topic Network for providing a good global overview of large-scale image collections; Interactive Visualization for capturing the users’ query intentions easily and effectively; Interestingness Matching for recommending the most relevant images according to the users’ intentions.

3. Topic Network Construction What can topic network represent? Global overview of large-scale image collections; Context among the topics; local contexts + global contexts Good navigation or browsing structure for accessing large-scale image collections.

3. Topic Network Construction Major steps for concept network construction Natural language processing for entity extraction; Word frequency estimation; Keyword extraction; Co-occurrence for context determination.

3. Topic Network Construction Social Tags & Discussions in Texts Social Tag Cleansing Topics & Topic Correlations Topic Network

3. Topic Network Construction Social Tags & Discussions in Texts Social Tag Cleansing Keyword Extraction Correlation Analysis Topics & Topic Correlations

3. Topic Network Construction Social Tags & Discussions in Texts Social Tag Cleansing Spam tags Noisy tags ……….. Topics & Topic Correlations

3. Topic Network Construction Social Tags & Discussions in Texts Keyword Extraction Numbers of images Numbers of discussions Numbers of clicks ………. Topics & Topic Correlations

3. Topic Network Construction Co-Occurrence Probability Semantic Similarity Distribution Similarity

3. Topic Network Construction Context between concepts

3. Topic Network Construction

3. Topic Network Construction

4. Topic Network Visualization Interestingness of topic self-significance: numbers of images, discussions & clicks Link-significance: numbers of linkages

4. Topic Network Visualization Using interestingness to highlight image topics

4. Topic Network Visualization Constraint-based topic clustering

4. Topic Network Visualization Constraint-based topic clustering

4. Topic Network Visualization Query-Driven Topic Network Visualization

4. Topic Network Visualization Query-Driven Topic Network Visualization

4. Topic Network Visualization Multidimensional Scaling

4. Topic Network Visualization

4. Topic Network Visualization Change of Focus

4. Topic Network Visualization Change of Focus---Poincare Mapping

4. Topic Network Visualization Change of Focus---Poincare Mapping

5. Interactive Query Specification Concept Network Navigation

5. Interactive Query Specification Personalized concept network generation

5. Interactive Query Specification

6. Most Representative Image Recommendation Image Content Representation

6. Most Representative Image Recommendation Image Similarity Characterization Color Histogram Kernel

6. Most Representative Image Recommendation Image Similarity Characterization Wavelet Filter Bank Kernel

6. Most Representative Image Recommendation Image Similarity Characterization Interest Point Matching Kernel

6. Most Representative Image Recommendation Image Similarity Characterization ---- mixture of kernels Multiple kernel learning for determining weighting parameters!

6. Most Representative Image Recommendation Kernel-Based Image Clustering

6. Most Representative Image Recommendation Support Vector Machine positive images negative images positive support vectors margin negative support vectors

6. Most Representative Image Recommendation Representativeness of image

6. Most Representative Image Recommendation Most Representative Image Selection Most representative images come from all these clusters and distribute according their coverage percentages; Images close to decision boundaries; Most frequent accessed images

6. Most Representative Image Visualization Kernel PCA Similarity-Preserving Image Projection

6. Most Representative Image Visualization Hyperbolic Image Visualization

6. Most Representative Image Visualization

6. Most Representative Image Visualization

6. Most Representative Image Visualization

7. User-Adaptive Image Recommendation User Intention Capturing

7. User-Adaptive Image Recommendation Personalized Interestingness of Image

7. User-Adaptive Image Recommendation

7. User-Adaptive Image Recommendation

Collaborative Filtering Social Rating Information User’s Personal History Potential Changes over time Social Network

Multi-Modal Deep Learning