1. 1 Visual Processing for Social Media Andrew C. Gallagher Tsuhan Chen September 30, 2012 Cornell University TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA

2. Outline  Social Media Overview  Visual Processing Overview  Social Media Insights Within the Image  Social Media Insights From Sharing 2 TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA

3. 3 Now, pictures of people  Examples of how social data has helped understand images of people  Some things I’ve learned about people from computer vision

4. 4 Understanding images of people

5. 5 What the computer sees

6. 6 Faces in the lab [Turk et al., Cog. Neuro. 1991] [Belhaumer et al., PAMI 1997] [Wiskott et al., PAMI, 1997] [Lucey et al., IJCV 2007] [Blanz et al., PAMI 2003] [Kanade, Kyoto U. 1973]

7. 7 Faces in the wild [Naaman et al., JCDL 2005] [Chen et al., IJIG 2002] [Zhang et al., ACM MM, 2003] [Berg et al., CVPR 2004] [Huang et al., UMASS 2007] [Everingham et al., BMVC, 2006] [Davis et al., ACM MM, 2005] But what about social data?

8. 8 Is this a family?

9. 9 The Loop Images and Computer Vision What we know about people

10. 10 Understanding Images of People  Describe people: How tall? How old?  Identify people: Who?  Why are they together?  Exploit the same context humans use!

11. 11 Understanding Images of People  Capture Context  Social Context July 2, 2005 8:27 PM Lat: 42.2902 Long: 85.5361 June 25, 2005 10:50 AM Lat: 42.3202 Long: 85.1261

12. 12 Understanding Images of People  Capture Context  Social Context Adult male height:177 cm Adult female height: 163 cm MLE mother-child:27 years MLE husband-wife:2 years MLE |sibling-sibling|:6 years

13. 13 What is social context? Social Context: information about people and their society that is useful for understanding images.  Distributions of ages and genders in social groups  Social relationships  Face position in a group image  First name popularity over time  Anthropometric measurements

14. 14 Group Images  What we know and learn about people:  Group dynamics  Computer vision task:  Measuring age, gender, of each person in a group

15. 15 Images of Groups  Identify age and gender  Recognize certain group events  Consider context and appearance [A. Gallagher, T. Chen, CVPR 2009]

16. 16 Contextual Features  Absolute face position  Size, position relative neighbor and group  Minimal spanning tree degree

17. 17 Evidence of Social Context  Relative positions of nearest neighbors depends on the social relationship  Mean distance is 306 mm NeighborsMale to FemaleOther to Baby

18. 18 Evidence of Social Context  Position carries demographic information  Female more likely near image center  Children more likely near image bottom

19. 19 Evidence of Social Context  Samples of faces based on image location Random samples

20. 20 Use All Context  5080 images with 28,231 faces  Classification improves with more contextual features

21. 21 Appearance Features  Project face into Fisher Space  Nearest neighbor density estimation Gender subspace Nearest neighbors

22. 22 Gender Estimation ContextAppearanceCombined

23. 23 Gender Estimation ContextAppearanceCombined

24. 24 Context and Appearance  Context contributes more when appearance is weak. ContextAppearanceCombined All Faces Small Faces

25. 25 Context for Scene Geometry Find the face vanishing line Estimated horizon from face positions Manually labeled horizon

26. 26 Context for Dining Event  Group Structure = Activity

27. 27 Row Segmentation [A. Gallagher, T. Chen, ICME 2009]

28. 28 Row Segmentation

29. 29 Row Segmentation

30. 30 Row Segmentation

31. 31 Social Relationship Retrieval Spouses Mother-Child [G. Wang, A. Gallagher, J. Luo, D. Forsyth, ECCV 2010]

32. 32 Names as Context  What we know and learn about people:  Government census data  Computer vision task:  Matching names to faces. Guessing age and gender.

33. 33  First names capture information about age and gender  First names are social context Person A and Person B First Names as Context Mildred and Lisa Source: U.S. Social Security Administration [A. Gallagher, T. Chen, CVPR 2008]

34. 34 First names and appearance? Tom_101Ben_165Caleb_337Andrew_233Brian_116Zachary_431 1953 1956200319841962 1996 Abigail_194Heather_224Alejandra_152Juanita_192Ethel_165Gertrude_53 2002 1970197719471926 1924

35. 35 Gertrude_53 1924 Sort by Expected Age Tom_101 1953 Ben_165 1956 Caleb_337 2003 Andrew_233 1984 Brian_116 1962 Zachary_431 1996 Abigail_194 2002 Heather_224 1970 Alejandra_152 1977 Juanita_192 1947 Ethel_165 1926

36. 36 First Names as Context Mildred and Lisa Name Birth Year Age Features Gender Features Image-Based Features

37. 37 More Context = Better Results AppearanceFirst NameFull Model

38. 38 Recognition from First Name  The model improves name assignment, age estimation, and gender classification

39. 39 Learning about people 39 Images and Computer Vision What we know about people

40. 40 Group Images  How close do people stand in group photos?  Computer vision answer: 306 mm  Sociology’s “Personal Space”: 457 mm  Do people suspend personal space needs during photograph?

41. 41 Group Images: Gender Prior  How do people end up in a group photo anyhow?

42. 42 Group Images: Gender Prior  Bernoulli world?  Implicit prior, IID:  Let’s look at the data!

43. 43 “Group Shots” Number of Females Gender Distribution of 6 people Binomial Distribution Number of Females ? ? “Family” Actual Distributions Genders of people in a image are not independent!

44. 44 Group Shot Analysis  Standing Order Frequency for 4 people (2 male, 2 female): 0.13 0.11 0.19 0.13 0.30 0.15 But why?

45. 45 Learning about people 45 Images and Computer Vision What we know about people (what they do and think!)

46. 46 Social Context Data Summary  U.S. Social Security First Name Database  6693 first names, birth years, gender  U.S. CDC National Center for Health Statistics  Physical growth tables  Birth rates and other birth statistics  Family structure statistics  Farkas, 1994  Facial anthropometric measurements

47. 47 Conclusions  Social context is useful for interpreting single images or image collections  Social context is learned from images or other public sources  Learning about people improves our understanding of images of people

48. 48 Related Publications J. Xin, A. Gallagher, L. Cao, J. Luo, J. Han The Wisdom of Social Multimedia: Using Flickr For Prediction and Forecast ACM MM 2009 G. Wang, A. Gallagher, J. Luo, D. Forsyth Seeing People in Social Context: Recognizing People and Social Relationship ECCV 2010 A. Gallagher, A. Blose, T. Chen Jointly Estimating Demographics and Height with a Calibrated Camera ICCV 2009 A. Gallagher, T. ChenUsing Context to Recognize People in Consumer ImagesIPSJ Trans. on Comp. Vis. and Apps., 2009 A. Gallagher, T. ChenUnderstanding Images of Groups of PeopleCVPR 2009 A. Gallagher, T. ChenFinding Rows of People in Group ImagesICME 2009 A. Gallagher, C. Neustaedter, J. Luo, L. Cao, T. Chen Image Annotation Using Personal Calendars as ContextACM MM 2008 A. Gallagher, T. ChenEstimating Age, Gender and Identity using First Name Priors CVPR 2008 A. Gallagher, T. ChenClothing Cosegmentation for Recognizing PeopleCVPR 2008 P. Singla, H. Kautz, J. Luo, A. Gallagher Discovery of Social Relationships in Consumer Photo Collections Using Markov Logic CVPR SLAM 2008 A. Gallagher, T. ChenUsing a Markov Network to Recognize People in Consumer Images ICIP 2007 A. Gallagher, M. Das, A. Loui User-Assisted People Search in Consumer Image Collections ICME 2007 A. Gallagher, T. ChenUsing Group Prior to Identify People in Consumer ImagesCVPR SLAM 2007