1. Web Image Prediction Using Multivariate Point Processes Gunhee Kim 1 Li Fei-Fei 2 Eric P. Xing 1 1 1 : School of Computer Science, Carnegie Mellon University 2 : Computer Science Department, Stanford University August 14, 2012

2. Problem Statement Method  Multivariate Point Process + Poisson Regression  Full model of Intensity Function  Learning and Prediction  Personalization Experiments Conclusion Outline 2

3. Problem Statement Method  Multivariate Point Process + Poisson Regression  Full model of Intensity Function  Learning and Prediction  Personalization Experiments Conclusion Outline 3

4. 4 Problem Statement - Web Image Prediction A photo stream of world+cup from Flickr up to 12/31/2008. Each image is associated with meta-data (timestamp, owner ID). Can we guess what photos will appear on the Flickr at t q = 6/6/2009? Actual images at t q Collective Image prediction Actual images by u q at t q Personalized Image prediction

5. 5 Why is Image Prediction Interesting? Predicting User Behaviors on the Web User behavior on the Web changes over time. (2) News recommendation (3) Product search Few previous work on what images people are interested in. [D08] Dakka et al. CIKM 2008 [M09] Metzler et al. SIGIR 2009 [K10] Kulkani et al, WSDM 2011 [V11] Amodeo et al, CIKM2011 [R12] Radinsky et al, WWW 2012 What query terms are popular? (1) Keyword search What documents are most relevant? What documents are likely to be clicked?

6. Why is Image Prediction Interesting? Time-sensitive Image Reranking Submit the term world+cup into Google/Bing/Flickr engines Google Bing Flickr Severely redundant. Almost identical all year long. Any meaningful order? Increase diversity by temporal trends Ranking by temporal suitability

7. Why is Image Prediction Interesting? Time-sensitive Image Reranking Time-sensitive image reranking For t q = Jun. 23 (summer) For t q = Feb. 5 (winter) Personalized Time-sensitive image reranking For t q = Aug. 23 and u q = 15655191

8. 8 Relation to Previous Work Web Content DynamicsSimilar Image Retrieval Image based Collaborative Filtering Leveraging Web Photos to Infer Missing Information Text based method [A11,W06] Image-based method [K10] No image prediction No personalization Temporal trends + user histories Semantic meaning of keyword + feature-wise similarity [D11, P08, T08] Social trends in politics and market [J10] Spatio-temporal events [S10] Scene completion [H07] 3D models of landmarks [SN10] Semantic image hierarchy [L10] Images: source of prediction not subject of prediction Future images: not studied as missing info to be inferred. [A11] Ahmed et al. AISTAT11 [W06] Wang et al. KDD06 [K10] Kim et al, ECCV10 [D11] Deng et al. CVPR 11 [P08] Dhilbin et al. CVPR08 [T08] Torralba et al. PAMI08 [J10] Jin et al. MM10 [S10] Singh et al. MM10 [H07] Hayes et al. SIGGRAPH07 [SN10] Snavely et al. IEEE10 [L10] Li et al. CVPR10

9. 9 Summary of Contribution (2) News recommendation Collective and Personalized Web Image Prediction Algorithm based on multivariate point process (1) Predicting user behaviors on the Web (2) Time-sensitive image reranking Few previous work for large-scale Web images. Novel in image retrieval literature Flexibility, optimality, scalability, and prediction accuracies More than 10 million images of 40 topics Outperform baselines (PageRank based IR, Topic modeling)

10. Problem Statement Method  Multivariate Point Process + Poisson Regression  Full model of Intensity Function  Learning and Prediction  Personalization Experiments Conclusion Outline 10

11. 11 Multivariate Point Process (MPP) A stochastic process that consists of a series of random events in time and spaces. Neural spiking modeling [Brown et al. Nat.Neuro.04] Locations of Lauraceae trees [Moller et al. 2008] Ecology Computer Vision Crowd counting [Ge et al.CVPR08] Events in video [Prabhakar et al. CVPR10] Micro-earthquake data [Schoenberg] Statistical Model for spatio- temporal events Geology

12. 12 MPP for Image Streams An occurrence of a particular image at a particular time A short stream of penguin images Each image is associated with (visual cluster, timestamp) A point in time and image space = v 1 : ice hockey v 2 : animal penguin v 3 : snowy mountain Discrete-time trivariate PP

13. 13 Mathematical Formulation for MPP A short stream of penguin images Infinitesimal expected occurrence rate of visual cluster i at time t Intensity function for VC i at t The intensity function is represented by exponential of linear covariate functions. : Parameter set : covariate function Covariates: any likely factors to be associated with image occurrences (ex. Time, season, and other external events)

14. 14 MLE solution for MPP A short stream of penguin images Parametric form of intensity functions with covariates Log-likelihood of an observed stream MLE solution can tell which covariates are contributing for the occurrence of visual cluster i Poisson regression Globally-optimal solution

15. 15 Sparse MLE solution for MPP A short stream of penguin images Log-likelihood of an observed stream For each visual cluster, only a small number of strong factors affect image occurrence. A sparse solution is encouraged L1 (Lasso) penalty MLE solution: Cyclic coordinate descent [Friedman et al. 2010].

16. 16 A Toy Example of Image Prediction Covariates: only year and months (1 + 7 + 12 = 20 parameters) Shark example (Sea tour) (Ice hockey) Every year Observed occurrence data Peaked in summer Every month Peaked in January

17. Problem Statement Method  Multivariate Point Process + Poisson Regression  Full model of Intensity Function  Learning and Prediction  Personalization Experiments Conclusion Outline 17

18. 18 Full Model of Intensity Functions History component Correlation component External component Any probable factors can be included without performance loss because we encourage a sparse solution.

19. 19 Full Model of Intensity Functions History component Correlation component External component Linear autoregressive (AR) process of order P Typical pattern of annual periodicity Biphasic = bursty occurrence

20. 20 Full Model of Intensity Functions History component Correlation component External component Existence or absence of a VC can be a strong clue. Synchronized 4 months lag

21. 21 Full Model of Intensity Functions History component Correlation component External component Month covariateUser covariate Note 1. Flexibly add or remove covariate functions according to the characteristics of image topics. 2. AR can be replaced by a more general temporal model such as ARMA.

22. Problem Statement Method  Multivariate Point Process + Poisson Regression  Full model of Intensity Function  Learning and Prediction  Personalization Experiments Conclusion Outline 22

23. 23 Learning and Prediction LearningPrediction For each visual cluster (VC) i, 1. Figure out covariates for intensity function 2. Observe the actual occurrence of VC i 3. Compute MLE solution by using cyclic coordinate descent. Given a topic keyword and t q, 1. Gather covariates info for t q. 2. Compute intensity function for each VC i, 3. Sample L images according to O(MJT), only once offline O(MJ), for each t q M: No. of VCs J: No. of covariates T: No. of time steps 30 min (with soccer topic of 810K images) << 1 sec M: = 200, J = 118, T = 1,500

24. Problem Statement Method  Multivariate Point Process + Poisson Regression  Full model of Intensity Function  Learning and Prediction  Personalization Experiments Conclusion Outline 24

25. 25 Personalization Idea of locally-weighted Learning [Atkeson et al.97] Collective Image prediction Personalized Image prediction Each image is equally weighted For a user u 6 Each image is weighted according to the user similarity with u 6 Learning is more biased.

26. Problem Statement Method  Multivariate Point Process + Poisson Regression  Full model of Intensity Function  Learning and Prediction  Personalization Experiments Conclusion Outline 26

27. 27 Flickr Dataset 10,284,945 images of 40 topic keywords Ex. Soccer dataset Nations Places Animals Objects Activities Abstract Hot topics 7 groups Seasonal variation Zipf’s law

28. 28 Experimental Tasks Split the dataset into training/test sets Timeline 12/31/2008 2010 Training data + image DB Randomly pick t q ±1 days Positive test imagesL Predicted images Collective Image prediction Personalized Image prediction Randomly chosen 20 (t q,u q ) pairs Randomly chosen 20 t q per topic

29. 29 Evaluation Measures Actual images and predicted images are more then hundreds. How can we compare them? (1) Two distance metrics : Lower is better (2) Average precision: higher is better. L2Tiny [Torralba et al. 2008] SIFT/HOG 2 ******* ******* 2 Resize 32x32 images Using predicted images Rank positive/negative test images

30. 30 Quantitative Results Baselines 30 Sampling from ImageNet Semantic meaning only PageRank based IR Author-Time topic model State-of-the-art retrieval Generative topic model Collective Image predictionPersonalized Image prediction 7~8% higher than the best baseline.

31. 31 Examples of Collective Image Prediction World+cup (a) Jan. (b) May (c) Sep. Ski+skating Bicycle+kayak+soccer Soccer world cup Cardinals (a) Jan. (c) Sep. (b) May Football / Snow Baseball / Leafy, Eggs Baseball / Leafy

32. 32 Examples of Personalized Image Prediction Class Fine+art (a) User1 (b) User2 (c) User3 Painting Photography Flower Brazilian (a) User1 (c) User3 (b) User2Dance Auto-racing

33. Problem Statement Method  Multivariate Point Process + Poisson Regression  Full model of Intensity Function  Learning and Prediction  Personalization Experiments Conclusion Outline 33

34. 34 Conclusion Example code will be available ! What’s done Web image prediction (1) User behavior prediction (2) Time-sensitive image reranking Observations Poisson regression on multivariate point process Many topics are associated with predictable periodic events. Image-based Personalization is important. Ex. What styles of painting does user A like? More delicate information about user preference over texts