1. Page 1 Generalized Inference with Multiple Semantic Role Labeling Systems Peter Koomen, Vasin Punyakanok, Dan Roth, (Scott) Wen-tau Yih Department of Computer Science University of Illinois at Urbana-Champaign

2. Page 2 Outline System Architecture  Pruning  Argument Identification  Argument Classification  Inference [main difference from other systems] Inference with Multiple Systems  The same approach used by the SRL to assure a coherent output is used with input produced by multiple systems.

3. Page 3 System Architecture Identify argument candidates  Pruning  Argument Identifier Binary classification Classify argument candidates  Argument Classifier Multi-class classification  Inference Use the estimated probability distribution given by the argument classifier, and Expressive structural and linguistic constraints. Infer the optimal global output – modeled as a constrained optimization problem

4. Page 4 Pruning [Xue&Palmer 2004] Significant errors due to PP attachment Consider PP as attached to both NP and VP DevelPrecRecF1 Gold30.1996.5746.00 Charniak26.6185.4740.59

5. Page 5 Modified Pruning DevelPrecRecF1 Gold30.1996.5746.00 Charniak26.6185.4740.59 Charniak Modified heuristic 23.3187.5936.83

6. Page 6 Argument Identification Argument identifier is trained with a phrase-based classifier. Learning Algorithm – SNoW  A sparse network of linear classifiers Weight update: a regularized variation of the Winnow multiplicative update rule  When probability estimation is needed, we use softmax

7. Page 7 Argument Identification (Features) Parse tree structure from Collins & Charniak’s parsers Clauses, chunks and POS tags are from UPC processors

8. Page 8 Argument Classification Similar to argument identification, using SNoW as a multi-class classifier Classes also include NULL

9. Page 9 Inference Occasionally, the output of the argument classifier violates some constraints. The inference procedure [Punyakanok et al., 2004]  Input: the probability estimation (by the argument classifier), and structural and linguistic constraints  Output: the best legitimate global predictions  Formulated as an optimization problem and solved via Integer Linear Programming.  Allows incorporating expressive (non-sequential) constraints on the variables (the arguments types).

10. Page 10 Integer Linear Programming Inference For each argument a i  Set up a Boolean variable: a i,t indicating if a i is classified as t Goal is to maximize   i score(a i = t ) a i,t  Subject to the (linear) constraints Any Boolean constraints can be encoded this way. If score(a i = t ) = P(a i = t ), the objective is find the assignment that maximizes the expected number of arguments that are correct and satisfies the constraints

11. Page 11 Constraints No overlapping or embedding arguments  a i, a j overlap or embed: a i,NULL + a j,NULL  1

12. Page 12 Constraints  No overlapping or embedding arguments  No duplicate argument classes for A0-A5  Exactly one V argument per predicate  If there is a C-V, there must be V-A1-C-V pattern  If there is an R-arg, there must be arg somewhere  If there is a C-arg, there must be arg somewhere before  Each predicate can take only core arguments that appear in its frame file. More specifically, we check for only the minimum and maximum ids

13. Page 13 Results PrecRecF1 DevCollins73.8970.1171.95 Charniak75.4074.1374.76 WSJCollins77.0972.0074.46 Charniak78.1076.1577.11 BrownCollins68.0363.3465.60 Charniak67.1563.5765.31

14. Page 14 Inference with Multiple Systems The performance of SRL heavily depends on the very first stage – pruning [IJCAI 2005]  which is derived directly from the full parse trees Joint Inference allows improvement over semantic role labeling classifiers  Combine different SRL systems through joint inference  Systems are derived using different full parse trees

15. Page 15 Inference with Multiple Systems Multiple Systems  Train and test with Collins’ parse outputs  Train with Charniak’ best parse outputs Test with 5-best Charniak’ parse outputs

16. Page 16..., traders say, unable to cool the selling panic in both stocks and futures. a1a1 a1a1 a4a4 b1b1 b3b3 b2b2 traders the selling panic in both stocks and futures NullA0A1A2 0.20.40.2 NullA0A1A2 0.300.70 NullA0A1A2 0.10.20.40.3 NullA0A1A2 0.10.30.20.4 Naïve Joint Inference NullA0A1A2 0.3 0.2

17. Page 17 a1a1 a1a1 a4a4 a3a3 a2a2 b1b1 b3b3 b2b2 b4b4 NullA0A1A2 0.550.20.150.1 Joint Inference – Phantom Candidates Default Priors

18. Page 18 Results of Joint Inference

19. Page 19 Results of Joint Inference

20. Page 20 Results of Joint Inference

21. Page 21 Results of Different Combination

22. Page 22 Conclusion The ILP inference can naturally be extended to reason over multiple SRL systems.

23. Page 23 Thank You