1. Policy Gradient as a Proxy for Dynamic Oracles in Constituency Parsing
Training models that make local decisions to be aware of non-local consequences. I’ll be talking about two ways of dealing with this for constituency parsing: dynamic oracles (successful recently), and policy gradient (focus of this paper).
Daniel Fried and Dan Klein

2. Policy Gradient as a Proxy for Dynamic Oracles in Constituency Parsing
Training models that make local decisions to be aware of non-local consequences. I’ll be talking about two ways of dealing with this for constituency parsing: dynamic oracles (successful recently), and policy gradient (focus of this paper).
Daniel Fried and Dan Klein

3. Policy Gradient as a Proxy for Dynamic Oracles in Constituency Parsing
Training models that make local decisions to be aware of non-local consequences. I’ll be talking about two ways of dealing with this for constituency parsing: dynamic oracles (successful recently), and policy gradient (focus of this paper).
Daniel Fried and Dan Klein

4. Parsing by Local DecisionsVP NP NP
The
cat
took a
nap .
We’ll be working with transition-based parsers. These produce parse trees incrementally, conditioned on the sentence, using some decomposition of the tree. Variety of different ways to derive the tree, historically bottom-up shift reduce methods, but as a running example I’ll show a top-down method used successfully recently. It produces a linearized representation of the tree:
In all of the transition-based parsers we’ll look at, actions are predicted incrementally conditioned on the input sentence, and all of the past actions. Likelihood for a given sentence is the sum of log probs of these local decisions, in green.
Since these parsers make incremental predictions, they’re fast, and in recent years they’ve obtained state-of-the-art. But these local decisions may have non-local consequences. (S
(NP
The
cat )
(VP …
𝐿 𝜃 = log 𝑝 𝑦 𝑥;𝜃) = 𝑡 log 𝑝( 𝑦 𝑡 | 𝑦 1:𝑡−1 ,𝑥;𝜃)

5. Non-local Consequences
Loss-Evaluation Mismatch
The
cat
took a
nap . NP VP S
The
cat
took a
nap . VP NP S 𝑦 𝑦
∆(𝑦, 𝑦 ): -F1(𝑦, 𝑦 )
Exposure Bias
What are the consequences of producing structure using local decisions? Past work has identified a few potential problems.
Loss-evaluation mismatch. These parsers are trained to maximize the likelihood of each decomposed action. But this isn’t necessarily aligned with evaluation metrics we care about, which are defined on the entire structure in a way that doesn’t decompose along the actions.
Second, more subtle issue: exposure bias: each prediction conditions on the entire past history. The parser gets fed its own predictions. What if it makes a mistake, and now has some context that is unlike any it saw in training? Each mistake poisons your future decisions, unless you have a principled way of dealing with this in training.
True
Parse 𝑦 (S
(NP
The
cat …
Prediction 𝑦 (S
(NP
(VP ??
[Ranzato et al. 2016; Wiseman and Rush 2016]

6. Dynamic Oracle Training
Explore at training time. Supervise each state with an expert policy. 𝑦
True Parse (S
(NP
The
cat …
addresses exposure bias
Prediction
(sample, or greedy) 𝑦 (S
(NP
(VP
The …
Oracle
𝑦 ∗
(NP
The
The
cat
Dynamic oracles are a way to address both of these.
Have the model follow its own predictions at training time, and supervise with an expert, called a dynamic oracle. Design a dynamic oracle to tell you what’s the best action to take from any parser state that you could wind up in.
𝐿 𝜃 = 𝑡 log 𝑝( 𝑦 𝑡 ∗ | 𝑦 1:𝑡−1 ,𝑥;𝜃)
addresses loss mismatch
𝑦 𝑡 ∗
choose to maximize achievable F1 (typically)
[Goldberg & Nivre 2012; Ballesteros et al. 2016; inter alia]

7. Expert Policies / Dynamic Oracles PTB Constituency Parsing F1
Dynamic Oracles Help!
Expert Policies / Dynamic Oracles
Daume III et al., 2009; Ross et al., 2011; Choi and Palmer, 2011; Goldberg and Nivre, 2012; Chang et al., 2015; Ballesteros et al., 2016; Stern et al. 2017
mostly dependency parsing
PTB Constituency Parsing F1
System
Static Oracle
Dynamic Oracle
Coavoux and Crabbé, 2016
88.6
89.0
Cross and Huang, 2016
91.0
91.3
Fernández-González and
Gómez-Rodríguez, 2018
91.5
91.7
A long line of work shows that these dynamic oracles, or more generally expert policies, help improve performance.
Much of this work has been for dependency parsing, but in recent years people have successfully applied them to constituency parsing as well. They found consistent improvements.

8. What if we don’t have a dynamic oracle? Use reinforcement learning
Dynamic oracles are great if they apply – they give direct supervision that helps deal with non-local consequences. But they need to be custom-designed for a given parser’s transition system. In this work we’re going to investigate another, more general option, which you can use if you don’t have a dynamic oracle.
RL: discover non-local consequences: take some actions predicted by your model, observe their effect on the entire output, update accordingly.

9. Reinforcement Learning Helps! (in other tasks)
machine translation
Auli and Gao, 2014; Ranzato et al., 2016; Shen et al., 2016
Xu et al., 2016; Wiseman and Rush, 2016; Edunov et al. 2017
machine translation
RL has a long history, here are some nearest neighbors to our work: applying RL to other structured tasks in NLP.
CCG parsing
several, including dependency parsing

10. Policy Gradient Training
Minimize expected sequence-level cost:
True Parse 𝑦
Prediction 𝑦
𝑅(𝜃)= 𝑦 𝑝 𝑦 𝑥;𝜃 ∆(𝑦, 𝑦 )
The
man
had an
idea . NP VP S
The
man
had an
idea . NP VP S
∆(𝑦, 𝑦 )
𝛻𝑅 𝜃 = 𝑦 𝑝 𝑦 𝑥;𝜃 ∆ 𝑦, 𝑦 𝛻 log 𝑝( 𝑦 |𝑥;𝜃)
Can be applied across tasks with structured costs – directly address the loss-mismatch problem. Formally, minimize this risk objective: the expectation under the model of the structured cost.
Compute the gradient of this risk objective, it comes out to be this: an expectation of the gradient, scaled by the structured cost.
We have all of the parts that we need to compute this, and they’ll address both of the issues.
addresses exposure bias (compute by sampling)
addresses loss mismatch (compute F1)
compute in the same way as for the true tree
[Williams, 1992]

11. Policy Gradient Training
𝛻𝑅 𝜃 = 𝑦 𝑝 𝑦 𝑥;𝜃 ∆ 𝑦, 𝑦 𝛻 log 𝑝( 𝑦 |𝑥;𝜃)
Input, 𝑥
The cat took a nap.
The
cat
took a
nap . NP VP S
The
cat
took a
nap . NP VP
S-INV
The
cat
took a
nap . NP
ADJP S
The
cat
took a
nap . NP VP S
k candidates, 𝑦
∆(𝑦, 𝑦 )
(negative F1)
−89
−80
−80
−100 ∗ ∗ ∗ ∗
gradient for candidate
𝛻 log 𝑝( 𝑦 1 |𝑥;𝜃)
𝛻 log 𝑝( 𝑦 2 |𝑥;𝜃)
𝛻 log 𝑝( 𝑦 3 |𝑥;𝜃)
𝛻 log 𝑝(𝑦|𝑥;𝜃)

12. Experiments

13. Setup x Parsers Training Span-Based [Cross & Huang, 2016]
Top-Down [Stern et al. 2016]
RNNG [Dyer et al. 2016]
In-Order [Liu and Zhang, 2017]
Training
Static oracle
Dynamic oracle
Policy gradient x
We train these four parsers, each with a different transition system for decomposing the tree. We compare the three training methods we’ve described. Static oracle: standard, no exploration. Dynamic oracle exploration, if an oracle has been defined for the parser’s transition system. And policy gradient.
We also compare to another way of addressing loss-mismatch, based on softmax margin, see the paper for details.

14. English PTB F1 PG improves on static oracle in every setting.
We also compare to dynamic oracle training. Two of these parsers have existing dynamic oracles. One problem is you may not always have them. For the sake of comparison, we created a new dynamic oracle for RNNG in this paper. We find that it’s effective at supervising exploration. But you don’t have one until you invent it, and in-order doesn’t have one published yet.
The dynamic oracle is sometimes meaningfully better than policy gradient, sometimes worse. Both of them result in improvement.

15. PTB learning curves for the Top-Down parser
Training Efficiency
PTB learning curves for the Top-Down parser
Real advantage of the dynamic oracle. It can be a lot faster. Here’s a learning curve, showing development accuracy by training epoch, or number of passes through the training data.

16. French Treebank F1
Whether the dynamic oracle is better varies, but policy gradient is better than static oracle in almost all cases.

17. Chinese Penn Treebank v5.1 F1
Gains from both policy gradient and dynamic oracle.

18. Conclusions Local decisions can have non-local consequences
Loss mismatch
Exposure bias
How to deal with the issues caused by local decisions?
Dynamic oracles: efficient, model specific
Policy gradient: slower to train, but general purpose
More than before, we depend on local decisions. They have non-local consequences: training doesn’t directly optimize the cost you may be interested in, and the model may not be prepared to handle its own mistakes. We compared a couple methods for dealing with these issues. Dynamic oracles are effective, they work when they’re available, but they’re a knowledge-intensive solution. We find you can often get similar results with a more generic approach. Trade off training efficiency for agnosticism about the model. If you don’t have a dynamic oracle, you can just use policy gradient – it’s simple and typically effective.

19. Thank you!

20. For Comparison: A Novel Oracle for RNNG
(NP
The
man )
(VP
had …
1. Close current constituent if it’s a true constituent… (S
(NP
The
man )
… or it could never be a true constituent. (S
(VP
(NP
The
man ) )
Things that it checks to determine what next action to produce. Along with some examples.
2. Otherwise, open the outermost unopened true constituent at this position. (S
(NP
The
man )
(VP
3. Otherwise, shift the next word. (S
(NP
The
man )
(VP
had