Ranking and Classifying Attractiveness of Photos in Folksonomies Jose San Pedro and Stefan Siersdorfer University of Sheffield, L3S Research Center WWW 2009

2 Outline Introduction Introduction Features for Image Attractiveness Features for Image Attractiveness Experiment Data Experiment Data Experiment Results Experiment Results Conclusion Conclusion

3 Outline Introduction Introduction Features for Image Attractiveness Features for Image Attractiveness Experiment Setup Experiment Setup Experiment Results Experiment Results Conclusion Conclusion

4 Introduction Thousands of new photos are uploaded to Flickr every minute. Thousands of new photos are uploaded to Flickr every minute. –Effective automatic content filtering is necessary. Meta data for Flickr photos Meta data for Flickr photos –Tags –Number of views –User comments –Upload date –Save as favorite

5 Attractive or Not? Attractiveness of images Attractiveness of images –A highly subjective concept –Semantic aspects are associated but not crucial People expression, picture composition … etc People expression, picture composition … etc –The artistic component is an important factor –Low-level features are shown to provide high correlation with the attractiveness

6 Example Figure: Attractive vs. Unattractive images. Each column represents the same semantic concept (animal, landscape, portrait, flower) Figure: Attractive vs. Unattractive images. Each column represents the same semantic concept (animal, landscape, portrait, flower)

7 Outline Introduction Introduction Features for Image Attractiveness Features for Image Attractiveness Experiment Setup Experiment Setup Experiment Results Experiment Results Conclusion Conclusion

8 Feature Types Visual features Visual features –Color-associated –Coarseness (sharpness) –Various color spaces are adopted RGB, YUV, HSL … etc RGB, YUV, HSL … etc Text features Text features –Tags

9 Color Features - 1 Brightness Brightness –To measure the intensity of light wave –YUV color space Saturation Saturation –To measure the vividness ( Illustration of YUV color space, Y=0.5)

10 Color Features - 2 Colorfulness Colorfulness –To measure the difference against grey

11 Color Features - 3 Naturalness ( 自然度 / 真實度 ) Naturalness ( 自然度 / 真實度 ) –To measure the degree of correspondence between images and human perception of reality –HSL (Hue-Saturation-Lightness) color space –Pixels with 20 ≦ L ≦ 80 and S ＞ 0.1 are grouped into 3 sets: ‘A – Skin’, ‘B – Grass’, ‘C – Sky’

12 Color Features - 4 Contrast Contrast –To measure the relative variation of luminance –RMS-contrast

13 Coarseness Feature Coarseness represents the degree of detail contained in an image. Coarseness represents the degree of detail contained in an image. The most commonly used metric: Sharpness The most commonly used metric: Sharpness –Be determined as a function of its Laplacian, normalized by the local average luminance where μ xy denotes the average luminance around pixel (x,y)

14 Indication of Tags

15 Outline Introduction Introduction Features for Image Attractiveness Features for Image Attractiveness Experiment Setup Experiment Setup Experiment Results Experiment Results Conclusion Conclusion

16 Experiment Setup Data from Flickr Data from Flickr –2.2M photos uploaded between June 1 and 7, 2007 –Among which 35,000 photos are with at least 2 favorite assignments –A random sample of 40,000 photos without any favorite assignment as the negative examples –The number of favorites are used as relevance values Attractiveness classification Attractiveness classification –Classifier: Support Vector Machine (SVMlight) Ranking by attractiveness Ranking by attractiveness –Regression model: Support Vector Regression

17 Outline Introduction Introduction Features for Image Attractiveness Features for Image Attractiveness Experiment Setup Experiment Setup Experiment Results Experiment Results Conclusion Conclusion

18 Classification Results - 1 Table: Classification results (BEP) of 500 “attractive/unattractive” training photos Table: Classification results (BEP) of 500 “attractive/unattractive” training photos

19 Classification Results - 2 Table: Classification results (BEP) of 8000 “attractive/unattractive” training photos Table: Classification results (BEP) of 8000 “attractive/unattractive” training photos

20 Classification Results - 3 Figure: Precision-recall curves for visual and textual dimensions and their combination (8000 training photos per class, numFav ≧ 5) Figure: Precision-recall curves for visual and textual dimensions and their combination (8000 training photos per class, numFav ≧ 5)

21 Ranking Result Evaluated by Kendall’s Tau-b Evaluated by Kendall’s Tau-b Table: Ranking using regression Table: Ranking using regression

22 Outline Introduction Introduction Features for Image Attractiveness Features for Image Attractiveness Experiment Setup Experiment Setup Experiment Results Experiment Results Conclusions Conclusions

23 Conclusions We have used favorite assignments in Flickr to obtain training data for attractiveness classification and ranking. We have used favorite assignments in Flickr to obtain training data for attractiveness classification and ranking. The best performance is achieved by combining tags and visual information. The best performance is achieved by combining tags and visual information. We plan to extend and generalize this work to consider various kinds of resources such as videos or text. We plan to extend and generalize this work to consider various kinds of resources such as videos or text.

24