1. Time-Sensitive Web Image Ranking and Retrieval via Dynamic Multi-Task Regression Gunhee Kim Eric P. Xing 1 School of Computer Science, Carnegie Mellon University February 6, 2013

2. Image Ranking and Retrieval Goal: Find the images for a given query ex. Cardinal Text-based image retrieval 2

3. Image Ranking and Retrieval ex. Cardinal northern_cardinal_glamour.jpg File name http://www.allaboutbirds.org/ guide/Northern_Cardinal/id Url 3 Text-based image retrieval Ambiguity and noise due to mismatch. Scalable and successful so far Goal: Find the images for a given query

4. Recent Image Ranking and Retrieval Various efforts to improve text-based image search User relevance feedback [Wang et al. CVPR 11] Text-based search by apple chosen by a user Reranking on visual features Pseudo-relevance feedback [Liu et al. CVPR 11] Human labeled training data [Yang et al. MM10] Image click data [Jain et al. WWW11] 4

5. 5 Time-Sensitive Image Ranking and Retrieval From experiments of 7.5 millions of Flickr images of 30 topics we found three good reasons … Discovery of temporal patterns of Web image collections [D08] Dakka et al. CIKM 2008 [M09] Metzler et al. SIGIR 2009 [K10] Kulkani et al, WSDM 2011 [V11] Amodeo et al, CIKM2011 [R12] Radinsky et al, WWW 2012 ….. No previous work using temporal info on image retrieval [Related work] Exploring temporal dynamics of Web queries Popular search keywords and relevant documents change over time. ex) Keyword search, Product search, News recommendation

6. 6 Why Time-Sensitive Image Retrieval? (1/3) 1. Knowing when search takes place is useful to infer users' implicit intents. Cardinal: (1) the red bird in America. Fall to Winter (Sep. ~ Feb.) Google Bing (2) Arizona cardinals (football) (3) St. Louis cardinals (baseball) Spring to Fall (Mar. ~ Oct.) Severely redundant. Almost identical all year long.

7. 7 Why Time-Sensitive Image Retrieval? (1/3) 1. Knowing when search takes place is useful to infer users' implicit intents. at May 4, 2009 at Feb. 7, 2009 Football Google Bing Our results baseball Diversity can make search interesting.

8. 8 Why Time-Sensitive Image Retrieval? (2/3) 2. Timing suitability can be used as a complementary attribute to relevance. Google Bing at May 4, 2009 at Feb. 7, 2009 Our results There are so many almost equally good images. Background: snow Background: Green Baby birds or eggs

9. 9 Why Time-Sensitive Image Retrieval? (3/3) 3. Temporal information is synergetic in personalized image retrieval. Louisville Men's College Basketball At Nov. 7, 2009 for user 30033302 Each user’ term usages are relatively stationary, and predictable once they are learned.

10. 10 Algorithm Regularized multi-task regression on multivariate point process Goal: Scalably learn temporal models for each topic keyword. Multi-task framework: allows multiple image descriptors. Several regularization schemes Personalization by locally-weighted learning

11. Thank you ! Stop by our poster! 11

12. 12 Multivariate Point Process Models Given a stream of hornet pictures up to T Clustering by descriptor 1Clustering by descriptor 2 Time t 1 t 2 t 3 t 5 t 6 t 7 t 9 t 10 1st descriptor (v1)(v1) 2nd descriptor (v2)(v2)

13. 13 Regularized GLM on Point Processes Given a stream of hornet pictures up to T Time t 1 t 2 t 3 t 5 t 6 t 7 t 9 t 10 Formulate a regression between occurrence rates and covariates. Covariates: any likely factors to be associated with image occurrence (ex. Time, season, and other external events) Compute sparse regularized MLE solutions For each visual cluster, we select only a small number of strong factors. (v1,v2)(v1,v2) (3, 2) (3, 2) (3, 2) (2, 1) (1, 3) (1, 4) (2, 3) (2, 1)

14. 14 A Toy Example of Image Reranking Peaked in summer (Aquarium) (Sea tour) (Ice hockey) Peaked in January Covariates: only year and months Stationary