1. Automatic Classification of Semantic Relations between Facts and Opinions Koji Murakami, Eric Nichols, Junta Mizuno, Yotaro Watanabe, Hayato Goto, Megumi Ohki, Suguru Matsuyoshi, Kentaro Inui, Yuji Matsumoto Aaron Michelony CMPS 245 May 3, 2011

2. Abstract They want to classify and identify semantic relations between facts and opinions on the Web. This will enable them to organize information on the Web. Recognizing Textual Entailment (RTE) and Cross-document Structure Theory (CST) are sets of semantic relations. They will expand on these. Japanese web pages.

3. Recognizing Textual Entailment (RTE) The task of deciding whether the meaning of one text is entailed from another text. A major task in the RTE Challenge is classifying the semantic relation between a Text (T) and Hypothesis (H) into o [ENTAILMENT] o [CONTRADICTION]: It is very unlikely that both T and H can be true at the same time. o [UNKNOWN]

4. Cross-document Structure Theory (CST) Developed by Radev (2000). Another task of recognizing semantic relations between sentences. An expanded rhetorical structure analysis based on Rhetorical Structure Theory (RST) (1988). A corpus of cross-document sentences annotated with CST relations has been constructed. 18 kinds of semantic relations in this corpus, including [EQUIVALENCE], [CONTRADICTION], [JUDGEMENT], [ELABORATION], [REFINEMENT]. CST was designed for objective expressions.

5. Example Semantic Relations Query: Xylitol is effective at preventing cavities. Matching sentences and output: o The cavity-prevention effects are greater the more Xylitol is included [AGREEMENT]. o Xylitol shows effectiveness at maintaining good oral hygiene and preventing cavities. [AGREEMENT] o There are many opinions about the cavity-prevention effectiveness of Xylitol, but it is not really effective. [CONFLICT]

6. Semantic Relations between Statements Goal: Define semantic relations that are applicable over both fact and opinions.

7. [AGREEMENT] Bi-directional relation where statements have equivalent semantic content on a shared topic. Example: o Bio-ethanol is good for the environment. o Bio-ethanol is a high-quality fuel, and it has the power to deal with the environment problems we're facing.

8. [CONFLICT] Bi-directional relation where statements have negative or contradicting semantic content on a shared topic. Example: o Bio-ethanol is good for our earth. o There is a fact that bio-ethanol further the destruction of the environment.

9. [EVIDENCE] Uni-directional relation where one statement provides justification or supporting evidence for the other. Example: o I believe that applying the technology of cloning must be controlled by law. o There is a need to regulate cloning, because it can be open to abuse.

10. [CONFINEMENT] Uni-directional relation where one statement provides more specific information about the other or quantifies the situations in which it applies. Example: o Steroids have side-effects. o There is almost no need to worry about side-effects when steroids are used for local treatments.

11. Recognizing Semantic Relations 1.Identify a [AGREEMENT] or [CONFLICT] relation between the Query and Text. 2.Search the Text sentence for cues that identify [CONFINEMENT] or [EVIDENCE]. 3.Infer the applicability of the [CONFINEMENT] or [EVIDENCE] relations in the Text to the Query.

12. Linguistic Analysis Tools: o For syntactic analysis, the dependency parser CaboCha, which splits the Japanese text into phrase-like chunks and represents syntactic dependencies between the chunks as edges in a graph. o The predicate-argument structure analyzer ChaPAS. o Modality analysis resources provided by Matsuyoshi et al. (2010), focusing on tense, modality and polarity.

13. Structural Alignment Consists of two phases: 1.Lexical alignment 2.Structural alignment Aligns chunks based on lexical similarity information, creating an alignment confidence score between 0.0 and 1.0, aligning chunks whose scores cross an empirically- determined threshold.

14. Structural Alignment Uses the following information: o Surface level similarity  Identical content words or cosine similarity. o Semantic similarity  Predicates: Check for matches in a predicate entailment database.  Arguments: Check for synonyms or hypernym matches in WordNet or a hypernym collection.

15. Structural Alignment Compare the predicate-argument structure of the query to that of the text and see if they are compatible. Example: o Agricultural chemicals are used in the field. o Over the field, agricultural chemicals are sprayed. Uses the following information: o # of aligned children o # of aligned case frames o # of possible alignments in a window of n chunk o predicates indicating existence or quantity, e.g., many few, to exist, etc. o Polarity of both parent and child chunks

16. Structural Alignment Use an SVM, train on 370 sentence pairs. Features: o Distance in edges in dependency graph between parent and child for both sentences o Distance in chunks between parent and child o Binary features indicating whether each chunk is a predicate or argument according to ChaPAS. o POS of first and last word in each chunk. o When the chunk ends with a case marker, the case of the chunk otherwise none. o Lexical alignment score of each chunk pair.

17. Relation Classification After structural alignment, do semantic relation classification. Uses an SVM. Features: o Alignments o Modality o Antonym: Identifies [CONFLICT]. o Negation o Contextual Cues: Can identify [CONFINEMENT] or [EVIDENCE] relations. "Because" and "due to" are typical for [EVIDENCE] and "when" and "if" are typical for [CONFINEMENT].

18. Evaluation 1.Retrieve documents 2.Extract real sentences that include major subtopic words 3.Reduce noise in data 4.Reduce search space by identifying sentence pairs and prepare pairs, which look feasible to annotate 5.Annotate corresponding sentences with [AGREEMENT], [CONFLICT], [CONFINEMENT], [OTHER].

19. Results Compare two different approaches: 1.3-class: Semantic relations are directly classed into [AGREEMENT], [CONFLICT] and [CONFINEMENT]. Cascaded 3-class: Semantic relations are first classified into [AGREEMENT] and [CONFLICT] and then, using context cues, are some of them reclassified into [CONFINEMENT].

20. Results

21. Error Analysis A big cause of incorrect classification is incorrect lexical alignment. o More resources needed, more effective methods needed. Most serious problem is the feature engineering necessary to find the optimal way of applying structural alignments or other semantic information to semantic relation classification.