11 Learning to Suggest Questions in Online Learning to Suggest Questions in Online Forums Tom Chao Zhou, Chin-Yew Lin, Irwin King Michael R. Lyu, Young-In Song, Yunbo Cao Chinese University of Hong Kong Microsoft Research Asia AT&T Labs Research August 11, San Francisco, USA

22 Learning to Suggest Questions in Online Background Motivation Related Work Experiments Our Approach Conclusions and Future Work

33 Learning to Suggest Questions in Online Background Online forum –Web application –Interactive, domain-specific –E.g. travel, sports, programming

44 Learning to Suggest Questions in Online Background Threads Each thread contains a discussion topic

55 Learning to Suggest Questions in Online Background Questions are focus –[Shrestha and McKeown 2004] Mining knowledge, Question-Answer pairs –[Cong et al. 2008][Bian et al. 2008] Question search –How is Orange Beach in Alabama? –Any idea about Orange Beach in Alabama? Limitation –Unware query only capture one aspect of a topic

66 Learning to Suggest Questions in Online Background Motivation Related Work Experiments Our Approach Conclusions and Future Work

77 Learning to Suggest Questions in Online Motivation Suggest semantically related questions –How is Orange Beach in Alabama? –Is the water pretty clear this time of year on Orange Beach? –Do they have chair and umbrella rentals on Orange Beach? –Topic: “Travel in Orange Beach” –beach, water, chair, umbrella, rental…

88 Learning to Suggest Questions in Online Motivation Benefits –Explore information needs from different aspects “Travel”: beach, water, chair, umbrella –Increase page views Enticing users’ clicks on suggested questions –Relevance feedback mechanism Mining users’ click through logs on suggested questions

99 Learning to Suggest Questions in Online Background Motivation Related Work Experiments Our Approach Conclusions and Future Work

10 Learning to Suggest Questions in Online Related Work Question search –Translation model [Jeon, Croft and Lee 2005][Duan et al. 2008] –Translation based language model [Xue, Jeon and Croft 2008] Question recommendation –MDL-based tree cut model [Cao et al. 2008] Differences –Fuse both lexical and latent semantic information –Utilizing interactive nature of online forums

11 Learning to Suggest Questions in Online Background Motivation Related Work Experiments Our Approach Conclusions and Future Work

12 Learning to Suggest Questions in Online Our Approach Document representation –Bag-of-words Independent Fine-grained representation Lexically similar –Topic model Assign a set of latent topic distributions to each word Capturing important relationships between words Coarse-grained representation Semantically related

13 Learning to Suggest Questions in Online Our Approach TopicTRLM –Topic-enhanced Translation-based Language Model

14 Learning to Suggest Questions in Online Our Approach TopicTRLM –q: a query, D: a candidate question –w: a word in query – : parameter balance weights of BoW and topic model –Jelinek-Mercer smoothing TRLM score: BoW LDA score: topic model

15 Learning to Suggest Questions in Online Our Approach TRLM –C: question corpus, :Dirichlet smoothing parameter –T(w|t): word to word translation probabilities Use of LDA K: number of topics, z: a topic

16 Learning to Suggest Questions in Online Our Approach Estimate T(w|t) –IBM model 1, monolingual parallel corpus –Questions are focus of forum discussions, questions posted by a thread starter (TS) during the discussion are very likely to explore different aspects of a topic Build parallel corpus –Extract questions posted by TS, question pool Q –Question-question pairs, enumerating combinations in Q –Aggregating all q-q pairs from each forum thread

17 Learning to Suggest Questions in Online Background Motivation Related Work Experiments Our Approach Conclusions and Future Work

18 Learning to Suggest Questions in Online Experiments Data set –Crawled from TripAdvisor –TST_LABEL: labeled data for 268 questions –TST_UNLABEL: 10,000 threads at least 2 questions posted by thread starters –TRAIN_SET: 1,976,522 questions,971,859 threads Parallel corpus to learn T(w|t) LDA training data Question repository Question detector –Labeled sequential pattern mining[Cong et al. 2008]

19 Learning to Suggest Questions in Online Experiments Data analysis Post level Forum discussions are quite interactive Power law # Threads# Threads that have replied posts from TS Average # replied posts from TS 1,412,141566,2561.9

20 Learning to Suggest Questions in Online Experiments Data analysis Question level 68.8% thread starters asked questions On average 2 questions are asked by thread starters in each thread Question is a focus of forum discussions # Threads# Threads TSs’ posts contain questions Average # questions in TSs’ posts 1,412,141971,8592.0

21 Learning to Suggest Questions in Online Experiments Word translation IBM 1: semantic relationships of words from semantically related questions LDA: co-occurrence relations in a question

22 Learning to Suggest Questions in Online Experiments Labeled question LDA performs the worst, coarse-grained TRLM > TR > QL TopicTRLM outperforms other approaches

23 Learning to Suggest Questions in Online Experiments Topics’ joint probability distribution –For each q, consider its first subsequent question q’ posted by the TS as relevant –For 10,000 q, LDA to infer the most probable topic, aggregate the counts of topic transitions –K * K topic transition matrix as ground truth –KL divergence, the smaller, the better

24 Learning to Suggest Questions in Online Background Motivation Related Work Experiments Our Approach Conclusions and Future Work

25 Learning to Suggest Questions in Online Conclusions and Future Work Summary –Propose a question suggestion application in forums –Propose a method to build parallel corpus of related questions –Propose TopicTRLM, which fuses lexical knowledge with latent semantic knowledge Future work –How to measure and diversify the suggested questions? –How question suggestion could help long query suggestion?

26 Learning to Suggest Questions in Online Thanks! Q & A

27 Learning to Suggest Questions in Online FAQ Q: Which tools do you use? A: –GIZA++ [Och and Ney 2003] train IBM model 1. –GibbsLDA++ [Phan, Nguyen and Horiguchi 2008] to conduct LDA training and inference. –Porter Stemmer to stem question words. –Stop word list by SMART system, but 5W1H were removed

28 Learning to Suggest Questions in Online FAQ Q: Which metrics do you use? A: Precision at Rank R –MAP: Mean average precision –MRR: Mean reciprocal rank –KL-divergence: Kullback-Leibler divergence

29 Learning to Suggest Questions in Online FAQ Q: How to tune parameters? A: We used 20 queries from TST_LABEL, and employ MAP to tune parameters

30 Learning to Suggest Questions in Online FAQ Q:Aligned monolingual questions A: –Has anyone had an experiences with the Eden Condos in Perdido Key? –Does anyone know how the beaches are there in Perdido key? –Can you go fishing right from the shore on Orange Beach? –What kinds of rods, and bait is needed for fishing down there?

31 Learning to Suggest Questions in Online FAQ Query likelihood language model using Dirichlet smoothing (QL)

32 Learning to Suggest Questions in Online FAQ Translation model using Dirichlet smoothing (TR)