A Deep Memory Network for Chinese Zero Pronoun Resolution Qingyu Yin, Yu Zhang, Weinan Zhang and Ting Liu Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, Harbin, China

1 Introduction

What is Zero Pronoun Zero Pronoun (ZP) The gap in the sentence Express the sentence Represent something that is omitted Certain entities

Introduction Goal of this paper ZP is ubiquitous in Chinese Recover the ZP in the sentence Entities ：Antecedent Noun Phrases ZP – Antecedents 我吃了一个苹果，<ZP> 很甜。 ZP is ubiquitous in Chinese Overt Pronoun 96%(English) 64%(Chinese)

A common way to ZP resolution Classification Problem ZP (我吃了一个苹果，<ZP> 很甜) Select a set of NP candidates Mention pair approach Classify for each pair ZP,NP1 （鸭梨） ZP,NP2 （苹果） … ZP,NPn （小明的书） ZP – NP2

Challenges of ZP resolution Overlook Semantic information Difficult to represent ZPs zero pronoun –overt pronouns No descriptive information Gender（男、女） Number（单数、复数） Represent gaps with some available components Context information <ZP> 很甜

Challenges of ZP resolution Overlook Semantic information Represent gaps with some available components Context information Potential candidate antecedents Only some subsets of candidate antecedents are needed Select importance candidates Utilize them to build up representations for the ZP Memory Network Use the importance of candidate explicitly

2 The approach

The approach -- ZP Semantic information is overlooked ZP has no descriptive information No actual content

The approach -- ZP Semantic information is overlooked Represent ZP ZP has no descriptive information No actual content Represent ZP contextual information <ZP> taste sweet. apples books

The approach -- ZP For ZP ZP-centered LSTM Employ two LSTM one to model the preceding context one to model the following context

The approach – Select NP Represent NP Average content Head word of an NP LSTM-based approach for modeling NP Content information Context information

The approach – Memory Network Memory: { r(np1), r(np2), r(np3), …, r(npn) } Select NP to fill in the gap (ZP)

The approach – Memory Network Memory: { r(np1), r(np2), r(np3), …, r(npn) } Select NP to fill in the gap (ZP)

The approach – Memory Network Memory: { r(np1), r(np2), r(np3), …, r(npn) } Select NP to fill in the gap (ZP)

The approach Get attention score for each NP candidate r(ZP), r(npi) feature vector ve(ZP,npi) si = tanh(W(r(ZP), r(npi) , ve(ZP,npi) )) For all the candidate NPs Add a softmax layer to gain the final attention score

3 Experimental results

Experimental Results Data set: Ontonotes 5.0 Experimental results R P Baseline： Chinese zero pronoun resolution: A deep learning approach. [C]. 2016 ACL. - Chen chen and Ng R P F Baseline system 51.0 51.4 51.2 Our approach (hop 1) 53.0 53.3 53.1 Our approach (hop 2) 53.7 54.0 53.9 Our approach (hop 3) 54.2 54.1 Our approach (hop 4) 54.4 54.7 54.3

Experimental Results Effectiveness of modeling ZPs and NPs R P F ZPContextFree 52.0 51.7 51.9 AntContextAvg 51.4 51.5 AntContHead 52.2 52.5 52.3 Our approach (hop 1) 53.0 53.3 53.1

Experimental Results Visualize Attention

4 Conclusion

Conclusion Effective memory network for modeling ZPs Future work contextual information ZP-centered LSTM candidate antecedents Modeling candidate antecedents multi-layer attention Memory network Future work Embeddings UNK embeddings Avoid Feature engineering

Thanks ! Q&A