1. Kateryna Tymoshenko , Alessandro Moschitti University of Trento
Assessing the Impact of Syntactic and Semantic Structures for Answer Passages Reranking
Kateryna Tymoshenko , Alessandro Moschitti
University of Trento

2. Outline Problem Motivation Framework Experiments Conclusions
Representation
Learning
Experiments
Conclusions

3. Problem Answer Passages Reranking
Select the right answer passages (AP) among the candidates retrieved by a seach engine
Q: 空海一体战的核心思想是什么?
S1. 美国防部报告阐释空海一体战核心思想和主要实施措施。
S2. 次年，美国智库战略与预算评估中心向国会高调推出了《空海一体战：全新作战思想的起点》研究报告，第一次完整展示了空海一体战的作战构想。
S3. 尽管如此，本文件的内容是直接选自《空海一体战构想》以及《2013财年总体实施计划》，并准确阐述了空海一体战的核心思想和主要实施措施。
S4. 相应地，空海一体战的中心思想是前所未有的联合一体化水平，空军和海军部队能发动网络化、一体化的深度攻击，干扰、摧毁和击败对手的反介入和区域拒止能力。

4. the eighth wonder of the world
Motivation
Factor
Similarity
Concepts’ relations
Example
Q: What sports stadium has been billed as “the eighth wonder of the world"?
AP: The Titans used to be called the Oilers and played in the dilapidated Astrodome; if this was the eighth wonder of the world, we live on a shabby little planet indeed.
Focus: sports stadium
Property
However, finding the focus word and its category may be difficult and error prone and most importantly does not allow for solving non-factoid questions.
Automatic feature engineering based on kernel method have been developed
This paper: The focus of Q is stadium, Astrodome has to identified as a stadium
the eighth wonder of the world
Astrodome
Property
Automatically feature engineering approach ?

5. Framework

6. Framework Method: (a) Representation : trees of Q/AP enriched with LD
[1] Design a pair of trees for the Q/AP pair
[2] Add links
Obtain connected structures of Q/AP pair
Capture the relatedness of Q/AP pair
(b) Learning
Model: L2R models SVMs (Structural Kernel)
Connect structures , which contain patterns useful for capturing the relatedness of Q/AP
Use YAGO, DBpedia and WordNet to match constituents from QA pairs
shallow syntactic trees, or deeper
Encoding relational information between Q and AP, i.e., links between words or constituents, is essential for improving passage reranking

7. Representation Shallow chunk-based tree (CH)
Dependency-phrase based tree (DT2)
Dependency-based tree (DT1)
Lexical-centered dependency tree(DT3)
design and compare several syntactic and semantic structural representations of pairs of short texts, including dependency- and shallow chunk-based tree representations.
CH: We represent a pair of short texts as two trees with lemmas at leaf level and their part-of-speech (POS) tags at the preterminal level. Preterminal POS-tags are grouped into the chunk nodes and the chunks are further grouped into sentences.
DT1:We structure the output of dependency parsers to design a new grammatical relation centered tree. This is a dependency tree altered so that grammatical relations become nodes. Lemmas and their POS tags are allocated at the leaf and the preterminal levels, respectively. Finally, we add ‘::’ and the first letter of the respective POS tags to the leaves, e.g., “world::n”.

8. Add links Category: Lexical relations(REL)
Matching lemmas
Question Focus-based relations(FREL)
Matching the question focus type with the type of NEs in the AP, tag REL-FOCUS-<QC>
Type Match(TM)
Semantic match using LD
Wikipedia-based REL(wikiREL)
Capture synonyms or different variants of the same name
Lexical relation(REL):Structural relations in both kinds of trees are encoded using the REL tag, which links the related structures in the two texts.
Question Focus-based relations (FREL):Semantic relations specific to QA can be derived from the question focus and category. These are encoded using the REL-FOCUS-<QC>tag, where <QC> is substituted with a question class in the specific examples. Question focus and AP chunks, which contain NEs of type compatible with the question class, are marked by prepending the above tags to their label

9. Type Match Relations : isSubclassOf / isa Algorithm :
Example : 𝑇 𝑒𝑛𝑡 : AP , T gen : Q , 𝐿𝐷 𝑑 : YAGO
Astrodome(anchor, in AP) yago:Reliant_Astrodome(entity) yago:wordnet_stadium_ (class) stadium (in Q)
Detect semantic relations between word sequences in two texts
Word sequence  class / entity in LD
Encoding relational information between Q and AP, i.e., links between words or constituents, is essential for improving passage reranking.
以前的工作：问题和句子之间只考虑了：string matching and question classifier,这缺点是coverage problem和the non-perfect coverage and accuracy of classifiers and named entity recognizers(NER).
Additionally, other attempts to use semantic matching, e.g., based on WordNet, have failed.
Our approach targets entities defined in LD, it highly increases coverage and at the same time avoids errors of classifiers.
In this work, we employ robust and accurate entity match algorithms we defined in [37].
We detect semantic relations between word sequences in two texts, Tent and Tgen.
Type Match (TM)
We extract knowledge about entities, classes and their relations from the DBpedia, YAGO and WordNet datasets available as LD.
算法：
输入：两句话，LD (knowledge source)，空的TM
Line 2 scans Tent for the anchors that refer to classes or entities in LDd;
Line 3 for each anchor, aent in Tent, it detects the URIs of the corresponding entities in LDd
Line 4 for each uri the algorithm extracts a list of types that generalize it along with their human-readable labels
Line 5-6: it iterates through chunks in Tgen and human-readable labels of types extracted in the previous step and checks them for string match
输出：TM (Type Match)
Detecting anchors and referent entities/classes
(getAnchors and getURIs)
Wikification tool
YAGO and DBpedia are aligned with Wikipedia pages ,e.g, yago:hasWikipediaUrl
Extracting generalization classes and their names
(getTypes)
rdf:type / rdfs:subClassOf
rdfs: label

10. Learning L2R models SVMs and Structural Kernel
ℎ 𝑥 = 𝑖 𝑎 𝑖 𝑦 𝑖 𝐾(𝑥, 𝑥 𝑖 )
𝐾(𝑥, 𝑥 𝑖 ) (kernel function): computes the similarity between two objects
Convolution Tree kernel
𝑇𝐾 𝑇 1 , 𝑇 2 = 𝑛 1 ∈ 𝑁 𝑇 𝑛 2 ∈ 𝑁 𝑇 2 ∆( 𝑛 1 , 𝑛 2 )
∆ 𝑛 1 , 𝑛 2 = 𝑖=1 |𝐹| 𝐼 𝑖 ( 𝑛 1 ) 𝐼 𝑖 ( 𝑛 2 )
∆ 𝑛 1 , 𝑛 2 : The number of common fragments rooted at the 𝑛 1 and 𝑛 2 nodes
Design a kernel function for (preference) reranking
𝑃 𝑘 <𝑎,𝑏>,<𝑐,𝑑> =𝐾 𝑎,𝑐 +𝐾 𝑏,𝑑 −𝐾 𝑎,𝑑 −𝐾(𝑏,𝑐)
𝐾 𝑎,𝑐 =𝑇𝐾 𝑄 𝑎 , 𝑄 𝑐 +𝑇𝐾( 𝐴𝑃 𝑎 , 𝐴𝑃 𝑐 )
Set of Q/AP pairs: {𝑎,𝑏,𝑐,𝑑}
Feature: structures kernel machines, e.g., SVMs, for automatic feature generation , given the above
If we use kernel functions, we do not need to represent objects with features and Thus we do not need to design features at all
K counts the number of common subtrees between two trees

11. Experiments——Setup Test and train dataset
TREC QA 2002/2003 , TREC13 , Answerbag
LD dataset
YAGO, WordNet , DBpedia
Feature Vectors
Term-overlap features
PTK over tree representations
Search engine ranking score
Learning Models: SVM-Light-TK
Pipeline
UIMA framework, OpenNLP , Stanford CoreNLP, Illinois chunker
Wikification tools
Wikipedia Miner (WM) , Machine Linking (ML)
QA metrics
Precision at rank , MRR , MAP
Significance tests: use paired two-tailed t-test for evaluating the statistical significance of the cross-validation experiment.

12. Experiments——Comparing Shallow and Deep Structures
On TREC 2002/2003
CH+V+FREL ≈ DT1+V+FREL
CH+V+FREL < the other dependency structures
In our intuition, this new outcome may be due to the performance of the dependency parser employed for preprocessing

13. Experiments——Measuring the Impact of LD Semantics
[1]. Dbpedia表现不如
YAGO或Wordnet;
[2]. LD在结构上有用.
VL: TM 特征向量表示
[1] 发现一：这些表的结果现实，所有的系统利用LD知识，除了DBpedia only,都超过 CH+V+REL，CH+V+REL+FREL baselines.
[2] 发现二：Note that CH+V+REL+wikiREL systems enriched with TM tags perform comparably to CH+V+REL+FREL, i.e., using question and focus classifiers, and in some cases even outperform it. Thus LD models can avoid the use of training data and language/domain specific classifiers.
[3] 发现三：这些表的结果显示，当使用yago / wordnet，获得结果最好。这是因为，这些资源large-scale, fine-grained ，每个类或实体上包含大量同义词，允许有一个高的覆盖率TM links. DBpedia 类的层次比较浅，类包含labels少，所以对TM 效果不好。
[4] 发现四：using different TM-Knowledge encoding strategies ， TMn,TMnd,TMnf,TMndf，结果小的变化。意味encoding LD information 比我们想象的要简单。
[5] 发现五： Finally, the last three lines of Tab. 4 show our attempt to encode LD information in a feature vector, i.e., VL. This refers to the number of TM matches between the Q and AP, for different types of TM. It is basically the unstructured version of our LD models. As it can be seen, V+VL only slightly improves V and CH+V+VL+REL+FREL is about 4 points absolute less than the best model using LD in structures, i.e., +TMNDF . This confirms that semantic knowledge requires to be used in syntactic structures
[6] 发现六： Thanks to LD our system achieves an accuracy of 36.59, which would allow it to be ranked 3rd in the official evaluation, i.e., higher than MultiText-system (accuracy=35.1) and below the systems of LCC and Singapore (68.5and 41.9, respectively). However, such top two systems used many handcrafted rules, resources and heuristics, which also prevent researchers to replicate them. In contrast, we generated features automatically, we do not design rules, and all our technology is already off-the-shelf, except for some missing components that we will make freely available to facilitate replicability of our results.

14. Experiments TREC13: Sentence Reranking 【1】与现在的最好系统对比：
The comparison of our models with the state-of-the-art results, reported by Tab. 8,
shows that our kernel-based rerankers outperforms the very recent best model, i.e., Wang and Nyberg (2015), by 2.17% absolute in MRR and 1.31% in MAP. Moreover, when we add LD information, the improvement in MAP increases to 2.73% absolute.

15. Conclusions Problem: Answer Passage Reranking Method:
Representation: using syntactic and semantic structured enriched with LD
Learning: L2R models SVMs and Structural Kernel
Finding:
Full syntactic dependencies can outperform shallow models also using external knowledge
It is necessary that Encode semantic features in syntactic structures and generate syntactic/semantic relational patterns between question and answer passage(to be used as features in the reranker)
In particular, YAGO, DBpedia and Word-Net are used to match constituents from QA pairs. Such matches are used to enrich semantic structures. The experiments with TREC QA and the above models also combining traditional feature vectors and the improved relational structures greatly outperform a strong IR baseline, i.e., BM25, by 101%, and previous state-ofthe-art reranking models, e.g., up to 16% in MAP. Differently from previous work, our models can effectively use semantic knowledge in statistical learning to rank methods.
It should be stressed that our experiments have shown that simply using semantic information as features (even if extracted from a powerful resource as LD) does not significantly improve BM25. It is really necessary to encode semantic features in syntactic structures and then generate syntactic/semantic relational patterns between question and answer passage (to be used as features in the reranker).
Simply using semantic information as features (even if extracted from a powerful resource as LD) does not significantly improve BM25.

16. References
Alessandro Moschitti, Efficient Convolution Kernels for Dependency and Constituent Syntactic Trees. In Proceedings of the 17th European Conference on Machine Learning(ECML 2006), Berlin, Germany, 2006.
Aliaksei Severyn and Alessandro Moschitti, Automatic Feature Engineering for Answer Selection and Extraction. In proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2013), Seattle, Washington, USA. October, 2013.
Aliaksei Severyn and Massimo Nicosia and Alessandro Moschitti. Building Structures from Classifiers for Passage Reranking. In proceedings of the Conference on Information and Knowledge Management (CIKM 2013), San Francisco, CA, US, Oct 27, 2013 – Nov
Kateryna Tymoshenko, Alessandro Moschitti and Aliaksei Severyn. Encoding Semantic Resources in Syntactic Structures for Passage Reranking. In proceedings of the European Chapter of the Association for Computational Linguistics (EACL 2014), Gothenburg, Sweden, April 2014.
Kateryna Tymoshenko, Alessandro Moschitti: Assessing the Impact of Syntactic and Semantic Structures for Answer Passages Reranking. In Proceedings of The 24th ACM International Conference on Information and Knowledge Management(CIKM 2015): , Melbourne, Australia, Oct , 2015.

17. 致谢
欢迎老师和同学提问！