1. Two Discourse Driven Language Models for Semantics
Haoruo Peng and Dan Roth
ACL-2016

2. A Typical “Semantic Sequence” Problem
Co-reference
“he” refers to whom? QA
Who was arrested? / Was Kevin arrested by police?
Event
What happened to Robert after the robbery?
“Understanding what could come next” is central to multiple natural language understanding tasks.
[Kevin] was robbed by [Robert], and [he] was arrested by the police.
First, let’s start with a motivating example …

3. Two Key Questions
How do we model the sequential nature of NL at a semantic level?
What do we mean by “a semantic level”?
Outcome: Semantic Language Models (SemLMs)
Quality evaluation of SemLMs
How do we use the model to better support NLU tasks?
Application to co-reference
Application to shallow discourse parsing

4. Frame-Based Semantic Sequence
[Kevin] was robbed by [Robert], and [he] was arrested by the police.
predicate
rob
predicate
arrest
sub
obj
sub
obj
Robert
Kevin …
the police he …
argument
argument
argument
argument
For the modelling problem, we choose the frame-based definition.
[Chambers and Jurafsky, 2008; Bejan, 2008; Jans et al., 2012]
[Granroth-Wilding et al., 2015; Pichotta and Mooney, 2016] …

5. What We Do Differently … … Infuse discourse information
[Kevin] was robbed by [Robert], but the police mistakenly arrested [him].
predicate
predicate
connective
Into the mix (between frames)
Also this is important … …
argument
argument
argument
argument

6. Two Different Sequences
[Kevin] was robbed by [Robert], but the police mistakenly arrested [him].
Frame-Chain (FC) Sequences
Entity-Centered (EC) Sequences
SRL
SRL
Shallow
Discourse Parsing
Rob.01
Arrest.01
sub
obj
but
sub
obj
Robert
Kevin
the police
Kevin
him
Co-reference
Two models for different usages
Rob.01
but
Arrest.01
EOS
Kevin
Rob.01-obj
but
Arrest.01-obj 6

7. Semantic Language Model (SemLM)
SemLM Units
Disambiguated Predicates or Disambiguated Predicates with argument role label
Discourse Markers including
Language Models
Semantic Sequences
Semantic Knowledge
Rob.01
Rob.01-obj
but
EOS

8. Language Model Implementations
N-gram (UNI, BG, TRI)
Skip-Gram (SG)
Continuous Bag-of-Words (CBOW)
Log-bilinear (LB)
An extension to cbow

9. Building SemLMs from Scratch
Semantic Role Labeling
Shallow Discourse Parsing
End-to-End Co-reference
A Large Collection
of Text Documents
20 Years NYT
1.8M Documents
Annotated
Documents
Illinois NLP Packages
FrameNet Mapping
Augment to Verb Phrases
Augment to Compound Verbs
Filter Rare Units
( >=20 times)
Add “UNK”
SemLM
Units
SemLM
Vocabulary

10. Building SemLMs from Scratch
Augment to Verb Phrases
+ preposition “take over” take.03(over)
+ negation “not like” like.01(not)
be + adj. “be happy” be.02(happy)
Augment to Compound Verbs
eat and drink “eat.01-drink.01” decide to buy “decide.01-buy.01”
Together
Actual Vocabulary
He doesn’t want to give up.
want.01(not)-give.08(up)

11. Building SemLMs from Scratch
20 Years NYT
1.8M Documents
Annotated
Documents
Illinois NLP Packages
FrameNet Mapping
Augment to Verb Phrases
Augment to Compound Verbs
Filter Rare Units
( >=20 times)
Add “UNK”
SemLM
Units
SemLM
Vocabulary
SemLM Training
Similar Settings as LM
Two Models (FC / EC)
Four Implementations
8 Different
SemLMs

12. Design Choices Generate a Probabilistic Model
V.S. script learning  Too Sparse
e.g. Narrative Schema [Chambers and Jurafsky, 2009]
Noise in Preprocessing (SRL, Co-ref, etc.)
Large Data  Robustness (Shown in Quality Evaluation)
Entity Modeling in Semantic Sequences
The right level of abstraction is hard to determine.
[The doctor] told [Susan] that [she] had been busy Person
[The doctor] told [Susan] that [she] had cancer Doctor/Patient
[Mary] told [Susan] that [she] had cancer ?

13. Quality of SemLMs Two Standard Tests Test Corpus Perplexity Test
Narrative Cloze Test
Test Corpus
NYT Hold-out Data (10% of NYT corpus) + Automatic Annotation
Gold PropBank Data with Frame Chains + Gold Annotation
Gold Ontonotes Data with Coref Chains + Gold Annotation

14. Quality of SemLMs
Perplexity Test

15. Quality of SemLMs
Narrative Cloze Test (MRR)
in the paper

16. Application to NLP Tasks
Co-reference Resolution
Mention-Pair Model
Conditional Probability
Add as additional features
Wiseman better results, but orthogonal

17. Application to NLP Tasks
Shallow Discourse Parsing
CoNLL Shared Task Setting (Connective Sense Classification)
Conditional Probability
Add as additional features

18. Conclusion
Thank You!
How do we model the sequential nature of NL at a semantic level?
SemLMs: Discourse Driven Two Models - Four Implementations High Quality
How do we use the modelling to better support NLU tasks?
Two Tasks, which utilize two models separately SemLM Conditional Probability as Features Trained Embeddings
SemLMs Available: me at

19. Example Output FC-LB EC-LB
P(convict.01 | arrest.01, indict.01) = 8.2×10-3
P(rescue.01 | arrest.01, indict.01) = 6.7×10-5
EC-LB
P(arrest.01-obj | rob.01-obj, but) = 2.1×10-3
P(arrest.01-obj | rob.01-sub, but) = 1.7×10-4
P(arrest.01-obj | rob.01-sub, and) = 7.3×10-3