1. INTRODUCING THE WEB INTELLIGENCE (WIT) GROUP Microsoft Research Asia

2. TALK OUTLINE  Introducing WIT – Web InTelligence Group  SQuAD  Summary

3. Mission Statement Enable synergetic collaboration between people and between people and computers to enlighten them and enrich their lives http://research.microsoft.com/en-us/groups/WIT/

4. Vision – a Web with Intelligence Satisfy user needs, simplify key tasks, promote serendipitous discovery, and foster task-oriented social network Web IntelligenceContent ReviewsForums… Action SearchBrowse… People FriendsExperts…

5. Web InTelligence group (WIT) I’m the manager! Chin-Yew Lin Tetsuya Sakai Yunbo Cao Wei Lai Bo Wang Youngin Song I’m the SECOND Japanese researcher at MSRA! I’m the FIRST Korean researcher at MSRA!

6. WIT spun off from the Natural Language Computing group in June 2009! I joined MSRA in April 2009! I joined MSRA in May 2009!

7. WIT research topics Social question answering and summarisation Sentiment analysis Expert and social search User intent/activity recognition and prediction Inarticulate user assistance Information access evaluation

8. TALK OUTLINE  Introducing WIT – Web InTelligence Group  SQuAD  Summary

9. Mining Community Knowledge: Social Q&A and Its Application Web Intelligence (WIT), Microsoft Research Asia Chin-Yew LIN cyl@microsoft.com

10. Search vs. Question Answering (QA)  Understanding what users want is difficult! User intention Search vs. Question Answering (QA)

11. QA Complements Search short queries long queries highmidlowhighmidlow Query50 4950 question1341229413611967 Total184172144185169117 short: length = 3 high: freq >100K, mid: between 1K and 50K, low: freq < 300

12.  Goal:  Create a scalable question and answering service  Methods:  Index all question and answer pairs (QnA) and their authors on the web  Enrich QnA through summarization  Expand QnA database by auto-posting questions to and acquiring answers from community QnA services  Refine QnA through Wiki-style online collaboration  Motivations:  Leverage and add value to search  Leverage questions that already have been answered  Leverage people’s knowledge and their networks Scalable Question Answering & Distillation

13. CampusCS

14. Baidu Zhidao ( 百度知道 )  17,012,767 resolved questions in two years’ operation.  8,921,610 are knowledge related.  96.7% of questions are resolved.  10,000,000 daily visitors.  71,308 new questions per day.  3.14 answers per question.  http://www.searchlab.com.cn ( 中国人搜索行为研究 /User Research Lab of Chinese Search) http://www.searchlab.com.cn

15. A Traditional QA Architecture A QA system gives direct answers to a question instead of documents Falcon QA system (LCC) Moldovan et al. ACL 2000 Surdeanu et al. IEEE Trans. PDS 2002 Best QA system in TREC 8 & 9 Average question answering time TREC 8: 48 seconds TREC 9: 94 seconds ModuleTREC8TREC9 QP 1.1% 1.2% PR (21.3 sec) 44.4% (24.9 sec) 26.5% PS 5.4% 2.2% PO 0.1% AP (23.4 sec) 48.7% (65.5 sec) 69.7% Falcon QA system module analysis: processing time Traditional IR

16. http://weblogs.hitwise.com/leeann-prescott/2006/12/yahoo_answers_captures_96_of_q.html Yahoo! Answers has 19,041,128 resolved questions in 26 categories adding about 48K questions per day. (August 24, 2007) Community Question and Answering

17. Community QnA in Details Context 2 Topic Context 1

18. topic Online Discussion Forum

19. FAQ About 28,424,184 results on Live Search using query: “FAQ travel” (Google: about 64,200,000) Context dependent

20. Question Mining Question Answer ing Question Utility Question Search & Recommendati on Question Gener ation Answer Summari zation Challenges

21. List of Papers Accepted  Recommending Questions Using the MDL-based Tree Cut Model – Cao et al.; WWW 2008  Searching Questions by Identifying Question Topic and Question Focus – Duan et al.; ACL 2008  Using Conditional Random Fields to Extract Contexts and Answers of Questions from Online Forums – Ding el al.; ACL 2008  Finding Question Answer Pairs from Online Forums – Cong et al.; SIGIR 2008  Question Utility: A Novel Static Ranking of Question Search – Song et al.; AAAI 2008  Answer Summarization: Understanding and Summarizing Answers in Community-Based Question Answering Services – Liu et al; COLING 2008  Automatic Question Generation from Queries – Lin; NSF Workshop on Question Generation Shared Task and Evaluation Challenge 2008

22. Question Mining & Answering (ACL 2008 & SIGIR 2008)  Extract question and answer pairs  Community QnA Create a resolved question list Extract & index question, best answer, and other answers Live Qna, Yahoo! Answers, Baidu Zhidao, …  Forum Extract and index threads and postings, find questions and their answers

23. QA Pairs in Online Forums

24. Question Search & Recommendation (ACL 2008 & WWW 2008)  Query  We would like to know what will be available to see in the Forbidden City because we understand that it will be under repairs.  Question search  Is it true that the Forbidden City is undergoing renovation & we won't be allow to enter?  Question recommendation  Would you get a lower price by not needing a guide for the Forbidden City and etc?  Can anybody recommend a budget hotel near Forbidden City?  Question = Topic + Focus + Others (TFO)  Search: same topic similar foci  Recommend: same topic different foci

25. Identifying Topic and Focus  Specificity: the inverse of the entropy of the topic term‘s distribution over the sub-categories  Order topic terms by their specificity Travel @Yahoo! Answers Asia Pacific Europe … China Japan … Travel @Yahoo! Answers Asia Pacific Europe … China Japan … China 1.Anyone know where to see the Dragon Boat Festival in Beijing? 2.Where is a good (Less expensive) place to shop in Beijing? 3.What's the cheapest way to get from Beijing to Hong Kong? Europe 1.How far is it from Berlin to Hamburg? 2.What is the cheapest way from Berlin to Hamburg? 3.Where to see between Hamburg and Berlin? 4.How long does it take from Hamburg to Berlin?

26. Question Utility (AAAI 2008)  Motivation  How useful is a question?  How should we rank questions without queries?  Definition  How likely a question would be asked again? The probability generating query Q’ from question Q (Relevance score) The prior probability of question Q reflecting a static rank of the question i.e. Question Utility

27. Answer Summarization (COLING 2008)  Example: “Where to stay in Paris?”  2,645 answers (Yahoo! Answers 03/04/09)  Is the “best answer” the best answer?  Question clustering  Find similar questions  Answer summarization  Aggregate answers for a question cluster Answer Taxonomy Question Taxonomy

28. Travel FAQ  Microsoft Travel Guide  Http://travel.msra.cn Http://travel.msra.cn

29. TALK OUTLINE  Introducing WIT – Web InTelligence Group  SQuAD  Summary

30. Knowledge Distillation & Dissemination Mixed Mode Scalable Question Answ ering and Distillation Highly Structured QnA FAQ Structured QnA QnA Semi-structured QnA Forum Unstructured QnA Web Knowledge Distillation and Dissemination Mixed Mode Question Answering

31. Q&A = Knowledge = Power  Q&A is complement to web keyword search  Q&A can enhance existing QnA and search services  Leverage existing knowledge in the question and answer forms and their authors  Acquire or elicit human knowledge automatically

32. Discussion