NTCIR /21 ASQA: Academia Sinica Question Answering System for CLQA (IASL) Cheng-Wei Lee, Cheng-Wei Shih, Min-Yuh Day, Tzong-Han Tsai, Tian-Jian Jiang, Chia-Wei Wu, Cheng-Lung Sung, Yu-Ren Chen, Shih-Hung Wu, Wen-Lian Hsu Academia Sinica, Taipei

2/21 Outline The Design Principal System Architecture Question Processing Passage Retrieval Answer Extraction Answer Ranking Performance Conclusion

3/21 The Design Principals of ASQA Reduce the cost by adopting existing components InfoMap: a knowledge representation framework Mencius: an NER engine AutoTag: a Chinese word segmentation tool Lucene: an open source IR engine SVMLight and opennlp.maxent : machine learning packages Minimizing system complexity Only shallow NLP techniques are used We want to see how a Chinese QA system performs without deep NLP techniques Incorporating human knowledge with machine learning methods Knowledge editing tool Knowledge as machine learning features Knowledge as dominant strategy

4/21 Chinese Word Segmentation Chinese text lacks explicit word boundaries. Word segmentation is a necessary step in many Chinese applications There are some word segmentation tools, but not designed for QA Combination rules are applied to form meaningful words for our QA system 第一 (Neu) 銅 (Na) 鐵 (Na) 公司 (Nc) First Copper Iron Corp. 第一銅鐵公司 第一 (Neu) 銅鐵 (Na) 公司 (Nc)

5/21 Architecture of ASQA SVM InfoMap Question Processing AutoTagMencius ME LuceneAutoTag Passage Retrieval Answer Ranking Mencius Filter word index char index documents Passages QType SegmentsQFocus, QLimitations Answer Candidates Answers Answer Extraction

6/21 Question Processing SVM InfoMap Question Processing AutoTagMencius ME LuceneAutoTag Passage Retrieval Answer Ranking Mencius Filter word index char index documents Passages QType SegmentsQFocus, QLimitations Answer Candidates Answers Answer Extraction

7/21 Question Processing Capture what the user want Question classification Goal: accurately classify a Chinese question into a question type Chinese Question: 奧運的發源地在哪裡？ Where is the originating place of the Olympics? Question Type: Q_LOCATION| 地 QFocus analysis Goal: Capture other detail information about the question such as QFocus, NE, Time, QFDescription

8/21 Taxonomy of Question Types

9/21 A Hybrid Approach for Chinese Question Classification Hybrid Approach SVM: machine learning binary classifiers for each question type InfoMap: knowledge representation framework syntactic templates for classifying questions Features for SVM QC model Character Character bigram HowNet Main Definition

10/21 A Hybrid Approach for Chinese Question Classification InfoMap and SVM are integrated according to their individual advantages The templates in InfoMap for matching question types are designed with high precision. The SVM model has the Hownet Main Definition semantic feature. It has better recall. Use InfoMap approach as the dominant strategy Only fallback to SVM if there is no InfoMap template matched

11/21 QFocus Analysis QFocus analysis is a tagging problem which is different from QType classification Some types of information are extracted by QFocus analysis QFocus: a QFocus is the category name of the answers Time (TI): Time or Date expressions Named Entities (NE): PERSON, LOCATION, ORGANIZATION QF Description (QFD): other description about the answer 請問 [2000 年 /TI] 的 [G8 高峰會 /NE] 在 [ 日本 /NE] 何地舉行 ? Year 2000 G8 summit Japan Which place in Japan hosted the G8 summit in 2000? 請問 [ 芬蘭第一位女總統 /QF] 為誰 ? Finland's first woman president Who is the Finland's first woman president? 請問 [2000 年 /TI] [ 沉沒於北極圈巴倫支海 /QFD] 的 [ 俄羅斯核子潛艇 /QF] 的名字 ? Year 2000 sank in the Barents Sea Russian nuclear submarine Which Russian nuclear submarine sank in the Barents Sea in 2000?

12/21 A Hybrid Approach of QFocus Analysis Combine syntactic rules and ME-model ME-model Tagging problem The ME Features are Context words, Context POS, Previous Tags 718 tagged question sentence Syntactic rules examples “Noun” string located behind “ 的 ”, “ 之 ”  QF “Noun” string located in front of “ 是 ”, “ 為 ”, “ 於 ”, and “ 在 ”  QF string quoted by “ 「」 ” and “( )“  QFD

13/21 Passage Retrieval with Lucene SVM InfoMap Question Processing AutoTagMencius ME LuceneAutoTag Passage Retrieval Answer Ranking Mencius Filter word index char index documents Passages QType SegmentsQFocus, QLimitations Answer Candidates Answers Answer Extraction

14/21 Passage Retrieval with Lucene The required operator Initial Query (IQ) sets quoted and noun terms as required Relaxed Query (RQ) doesn’t set any term as required The boosting operator Quoted terms: 2 Nouns: 1.2 Verbs:0.7 Q by IQ with W-idx Q by IQ with C-idx Sort Q by RQ with W-idx Q by RQ with C-idx Sort Any result? End NO YES Passage retrieval runtime workflow 請問台灣童謠「天黑黑」是由哪位作曲家所創作？ Initial query example: +" 作曲家 "^1.2 +" 台灣 "^1.2 " 創作 "^0.7 +" 童謠 "^1.2 +" 天黑黑 "^2 Relaxed query example: " 作曲家 "^1.2 " 台灣 "^1.2 " 創作 "^0.7 " 童謠 "^1.2 " 天黑黑 "^2 請問台灣童謠「天黑黑」是由哪位作曲家所創作？ Initial query example: +" 作曲家 "^1.2 +" 台灣 "^1.2 " 創作 "^0.7 +" 童謠 "^1.2 +" 天黑黑 "^2 Relaxed query example: " 作曲家 "^1.2 " 台灣 "^1.2 " 創作 "^0.7 " 童謠 "^1.2 " 天黑黑 "^2

15/21 Answer Extraction SVM InfoMap Question Processing AutoTagMencius ME LuceneAutoTag Passage Retrieval Answer Ranking Mencius Filter word index char index documents Passages QType SegmentsQFocus, QLimitations Answer Candidates Answers Answer Extraction

16/21 Answer Extraction Top 5 passages are sent to answer extraction module Named entity recognition (Mencius) PERSON, LOC, and ORG are recognized by ME-based NER engine Fined-grained and other coarse-grained types are identified by taxonomy and templates in InfoMap Answer filtering Answers which are incompatible with the QType are filtered out Compatibility of question and answer types is defined by a mapping table

17/21 Answer Ranking SVM InfoMap Question Processing AutoTagMencius ME LuceneAutoTag Passage Retrieval Answer Ranking Mencius Filter word index char index documents Passages QType SegmentsQFocus, QLimitations Answer Candidates Answers Answer Extraction

18/21 Answer Ranking Rank answer candidates with ranking scores A ranking score is calculated according to the QFocus analysis results NE Score QFocus Scores Cue Score

19/21 System Performance of CLQA Chinese to Chinese Task

Performance Fig.1 Fig.3 Fig.4 Fig.2

21/21 Conclusions We have demonstrated that an effective Chinese QA system can be created Shallow NLP techniques Integrating knowledge templates (InfoMap) and machine learning methods (SVM, ME) Open Source IR engine is usable for Chinese QA In the future work, we would like to include deeper NLP techniques Parsing Event structure/Relation