Prototype-Driven Grammar Induction Aria Haghighi and Dan Klein Computer Science Division University of California Berkeley

Grammar Induction DT NN VBD DT NN IN NN The screen was a sea of red

First Attempt DT NN VBD DT NN IN NN The screen was a sea of red

Central Questions How do we specify what we want to learn? How do we fix observed errors? What’s an NP? That’s not quite it!

Experimental Set-up Binary Grammar { X 1, X 2, … X n } plus POS tags Data WSJ-10 [7k sentences] Evaluate on Labeled F 1 Grammar Upper Bound: 86.1 XjXj X i XkXk

Experiment Roadmap Unconstrained Induction Need bracket constraint! Gold Bracket Induction Prototypes and Similarity CCM Bracket Induction PhrasePrototype NPDT NN PPIN DT NN

Unconstrained PCFG Induction Learn PCFG with EM Inside-Outside Algorithm Lari & Young [93] Results 0 i j n  (Inside)  (Outside)

Gold Brackets Periera & Schables [93] Result Constrained PCFG Induction

Encoding Knowledge What’s an NP? Semi-Supervised Learning

Encoding Knowledge What’s an NP? For instance, DT NN JJ NNS NNP Prototype Learning

Add Prototypes Manually constructed Grammar Induction Experiments

How to use prototypes? ? ? ? ? DT The NN koala VBD sat IN in DT the NN tree ¦¦ ? NP PP VP S

How to use prototypes? NP PP VP DT The NN koala VBD sat IN in DT the NN tree ¦ ¦ S JJ hungry ?NP

Distributional Similarity Context Distribution  (DT JJ NN) = { ¦ __ VBD : 0.3, VBD __ ¦ : 0.2, IN __ VBD: 0.1, ….. } Similarity  (DT NN)  (DT JJ NN)  (JJ NNS)  (NNP NNP) NP

Distributional Similarity Prototype Approximation  (NP) ¼ Uniform (  (DT NN),  (JJ NNS),  (NNP NNP) ) Prototype Similarity Feature span(DT JJ NN) emits proto=NP span(MD NNS) emits proto=NONE

Prototype CFG + Model NP PP VP DT The NN koala VBD sat IN in DT the NN tree ¦ ¦ S JJ hungry NP P (DT NP | NP) P (proto=NP | NP)

Prototype CFG+ Induction Experimental Set-Up BLIPP corpus Gold Brackets Results

Summary So Far Bracket constraint and prototypes give good performance!

Constituent-Context Model

Product Model Different Aspects of Syntax CCM : Yield and Context properties CFG : Hierarchical properties Intersected EM [Klein 2005] Encourages mass on trees compatible with CCM and CFG

Grammar Induction Experiments Intersected CFG and CCM No prototypes Results

Grammar Induction Experiments Intersected CFG + and CCM Add Prototypes Results

Reacting to Errors Possessive NPs Our Tree Correct Tree

Reacting to Errors Add Prototype: NP-POS NN POS New Analysis

Error Analysis Modal VPs Our TreeCorrect Tree

Reacting to Errors Add Prototype: VP-INF VB NN New Analysis

Fixing Errors Supplement Prototypes NP-POS and VP-INF Results

Results Summary

Conclusion Prototype-Driven Learning Flexible Weakly Supervised Framework Merged distributional clustering techniques with supervised structured models

Thank You!

Unconstrained PCFG Induction Binary Grammar { X 1, X 2, … X n } Learn PCFG with EM Inside-Outside Algorithm Lari & Young [93] 0 i j n  (Inside)  (Outside) V XjXj X i XkXk N XkXk XjXj V N