1 Introduction to Computational Linguistics Eleni Miltsakaki AUTH Fall 2005-Lecture 9

2 What’s the plan for today? Discourse models cont’d –DLTAG: Lexicalized Tree Adjoining Grammar for Discourse A DLTAG-based system for parsing discourse The Penn Discourse Treebank –

3 Basic references Anchoring a Lexicalized Tree-Adjoining Grammar for Discourse (1998), –B. Webber and A. Joshi What are Little Texts Made of? A Structural Presuppositional Account Using Lexicalized TAG –B. Webber, A. Joshi, A. Knott, M. Stone DLTAG System: Discourse Parsing with a Lexicalized Tree-Adjoining Grammar (2001) –K. Forbes, E. Miltsakaki, R. Prasad, A. Sarkar, A. Joshi and B. Webber The Penn Discourse Treebank (2004) –E. Miltsakaki, R. Prasad, A. Joshi and B. Webber

4 Motivation and basics of the DLTAG approach Discourse meaning: more than its parts Compositional vs non-compositional aspects of discourse meaning This distinction is often conflated in most of related work Smooth transition from sentence level structure to discourse level structure

5 The DLTAG view of discourse connectives Discourse connectives are treated as higher level predicates taking clausal arguments Basic types of discourse connectives: –Structural Subordinate conjunctions (when, although, because etc) Coordinate conjunctions (and, but, or) –“Anaphoric” Adverbials (however, therefore, as a result, etc)

6 Elements of LTAG  Initial and auxiliary trees  Initial: Encode predicate-argument dependencies  Auxiliary: recursive, modify elementary trees  anchors of elementary trees are semantic predicates  substitution and adjunction  D-LTAG is similar  anchors of elementary trees are semantic features which can be lexicalized with discourse connectives

7 D-LTAG Structures and Semantics Initial Trees (a) John failed his exam because he was lazy

8 Auxiliary trees (a) Mary saw John but she decided to ignore him. (b) Mary saw John. She decided to ignore him. 1. On the one hand, John loves Barolo. 2. So he ordered three cases of the ‘ On the other hand, he had to cancel the order 4. because he then found that he was broke.

9 Phenomena that DLTAG captures Arguments of a coherence relation can be stretched “long distance” Multiple discourse connectives can appear in a single sentence or even a single clause Coherence relations can vary in how and when they are realized lexically

10 Stretching arguments On the one hand, John loves Barolo. So he ordered three cases of the ’97. On the other hand, he had to cancel the order Because he then found that he was broke.

11 Non-Compositional Semantics Non-defeasible vs defeasible causal connection (a)The City Council refused the women a permit because they feared violence. (b)The City Council refused the women a permit. They feared violence. Presuppositional semantics (Knott et al, 1996): –Defeasible rule: When people go to the zoo, they leave their work behind. ( c) John went to the zoo. However, he took his cell phone with him.

12 DLTAG system for parsing discourse Theoretical framework: DLTAG Main system components: –Sentence level parsing –Tree extractor –Tree mapper –Discourse input representation –Discourse level parsing

13 Parser (Sarkar, 2000) –XTAG grammar –One derivation per sentence E.g. Mary was amazed

14 Tree extractor:identifying discourse units (a) While she was eating lunch she saw a dog

15 Tree mapper From sentence level structure to discourse structure

16 Discourse input representation

17 System Architecture

18 Example Discourse (a) Mary was amazed. (b) While she was eating lunch, she saw a dog. (c) She’d seen a lot of dogs, but this one was amazing. (d) The dog barked and Mary smiled. (e) Then, she gave it a sandwich

Derived and Derivation trees

20 Corpus example The pilots could play hardball by noting they were crucial to any sale or restructuring because they can refuse to fly the airplanes. If they were to insist on a low bid of, say $200 a share the board mightn’t be able to obtain a higher offer from the bidders because banks might hesitate to finance a transaction the pilots oppose. Also, because UAL chairman Stephen Wolf and other UAL executives have joined the pilots’ bid, the board might be able to exclude him for its deliberations in order to be fair to other bidders (Wall Street Journal) LEXTRACT (Xia et al 2000)

Corpus: Derivation Tree

22 Derived Tree

23 Summary points of the DLTAG system Implementation of D-LTAG  use LTAG grammar to parse each clause  use the same LTAG-based parser both at the sentence level and discourse level  build the semantics compositionally from the sentence to the discourse level  factor away non-compositional semantic contributions In the output representation  The semantics of the connectives form only part of the compositional derivation of discourse relations  Discourse connectives are NOT viewed as names of relations

24 The Penn Discourse Treebank  Annotation of discourse connective and their arguments  Large scale: annotation of the entire Penn Treebank (1 million words)

25 Merits of the PDTB  Discourse relations are lexically grounded Exposing a clearly defined level of discourse structure Enabling annotations with high reliability  Building on existing syntactic and semantic layers of annotation (Treebank, PropBank)  Annotations independent of the DLTAG (or any other) framework

26 Project description  Annotation of connectives in the Penn Treebank  30K tokens of connectives 20K explicit conns + 10K implicit conns  Annotation of ARG1 and ARG2 of conns Ex. Mary left early because she was sick. ARG1: Mary left early CONN: because ARG2: she was sick  Four annotators at the beginning, then two  To come: Semantic role labels for ARG1 and ARG2

27 Connectives  Subordinate conjunctions (when, because, although, etc.) ARG1 – ARG2 (1) Because [the drought reduced U.S. stockpiles], [they have more than enough storage space for their new crop], and that permits them to wait for prices to rise.

28 Connectives  Coordinate conjunctions (and, but, or, etc.) ARG1 – ARG2  (2) [William Gates and Paul Allen in 1975 developed an early language- housekeeper system for PCs], and [Gates became an industry billionaire six years after IBP adapted one of these versions in 1981].

29 Connectives  Adverbials (therefore, then, as a result, etc.) ARG1 – ARG2 (3) For years, costume jewelry makers fought a losing battle. Jewelry displays in department stores were often cluttered and uninspired. And the merchandise was, well, fake. As a result, marketers of faux gems steadily lost space in department stores to more fashionable rivals -- cosmetics makers.

30 Connectives  Implicit (annotators provide named expression for implicit connective) ARG1 – ARG2  (4) …[The $6 billion that some 40 companies are looking to raise in the year ending March 31 compares with only $2.7 billion raised on the capital market in the previous fiscal year]. IMPLICIT-(In contrast) [In fiscal 1984 before Mr. Gandhi came to power, only $810 million was raised].

31 Annotation guidelines   What counts as a connective? Including distinction between clausal adverbials and discourse adverbials  What counts as an argument? Minimally a clause  How far does the argument extend? Including distinction between arguments (ARG1 and ARG2) and supplements to arguments (SUP1 and SUP2 respectively) Interesting comparison with ProbBank annotations of verbs

32 WordFreak (T. Morton & J. Lacivita)

33 Comparison with the RST corpus RST-corpus 1.Higher-level annot. 2.Abstract discourse relations 3.Doesn’t contain the basis of the relations 4.Low inter-annotator agreement 5.Small scale (385 wsj files) 6.No explicit links to Treebank PDTB 1.Basic level annot. 2.Connectives+args 3.Relations anchored to lexical items 4.High inter-annotator agreement 5.Large scale(Treebank: 2,500 wsj files) 6.Links to Treebank and PropBank Interesting to see how RST labels relate to semantic role assignment in PDTB

34 Preliminary experiments  10 explicit connectives (2717 tokens) Therefore, as a result, instead, otherwise, nevertheless, because, although, even though, when, so that  386 tokens of implicit connectives  2 annotators

35 Inter-annotator agreement (1)  Measure by token (ARG1+ARG2) ARG1 and ARG2 counted together Total number of connective ARG1/ARG2 tokens = 2717  Agreement = 82.8% Subord. Conj. = 86% Adverbials = 57%

36 Agreement per connective (1) CONNECTIVESAGR No.Conn. Total% AGR When Because Even though Although So that % 88.2% 88.3% 81.8% 79.4% TOTAL SUBCONJ % Nevertheless Otherwise Instead As a result Therefore % 91.3% 61.0% 45.2% 78.6% TOTAL ADV OVERALL TOTAL %

37 Inter-annotator agreement (2)  Measure by ARG (ARG1, ARG2) Check agreement for ARG1 and ARG2 Total number of argument tokens = 5434 (2717 ARG ARG2)  Agreement = 90.2% –ARG1 = 86.3% –ARG2 = 94.1% –Subord. Conj. =92.4% –Adverbial: =71.8%

38 Agreement per connective (2) CONNECTIVES AGR No.Conn. Total% AGR When Because Even though Although So that % 93.4% 94.1% 90.1% 89.2% TOTAL SUBCONJ % Nevertheless Otherwise Instead As a result therefore % 95.7% 72.9% 65.5% 87.5 TOTAL ADV % OVERALL TOTAL %

39 Analysis of disagreement Majority of disagreement due to ‘partial overlap’: 79% (5) It was forced into liquidation before trial when investors yanked their funds after the government demanded a huge pre-trial asset forfeiture. DISAGREEMENT TYPENo.% Missing annotations No overlap % 5.6% PARTIAL OVERLAP TOTAL42279% Parentheticals Higher verb Dependent clause Other % 33.9% 34.1% 1.1% Unresolved101.9% TOTAL534100%

40 Reanalysis of agreement  Inter-annotator agreement counting in partial overlap 94.5%  Dealing with extent of the argument Revise guidelines BUT: Some disagreement will persist

41 Comparing predicates  PropBank – sentence level predicates (verbs) Arity of arguments: Hard Extent of the argument: Easy  Penn Discourse Treebank – discourse predicates Arity of arguments: Easy Extent of the argument: Hard

42 Summary points for PDTB   The Penn Discourse Treebank Large scale discourse annotation Basic level of annotation: connectives and their arguments Links to Penn Treebank and Penn PropBank (rich substrate for extracting syntactic and semantic features) Expected completion November 2005  Inter-annotator agreement Most conservative: 82.8% Relaxing exact match: 94.5%