Automatic Grammar Induction and Parsing Free Text - Eric Brill Thur. POSTECH Dept. of Computer Science 심 준 혁

2 CS730B Statistical NLP Abstract o Transformation-Based Approach for PS using the automatic induction of natural language grammar. Learning a set of ordered transformations which reduce parsing error parsing text into Syntactic B-tree with non-Terminals Unlabelled o Applied To 1) POS Tagging, 2) PP-Attachment, 3) Word Classification o Related Research Automatically acquiring phrase structure using distributional Analysis A transformation based approach to prepositional phrase attachment A simple rule-based part of speech tagger Eric Brill Dept. of Computer and Information Science University of Pennsylvania

3 CS730B Statistical NLP Contents Ê Introduction Ë Transformation-Based Error-Driven Learning Ì Learning Phrase Structure Í Experimental Results Î Conclusions

4 CS730B Statistical NLP 1. Introduction o New Approach for Grammar Induction Problem o Referenced Corpus : Penn Treebank, WSJ, ATIS corpus. o Merits m System Implementation Simplicity m Process Efficiency A small set of Transformation Rule A small set of Training Corpus m Relative Accuracy is good m Robust to noise or unfamiliar input ( than CFG-Based approach ) o Defects m Time Complexity in proportion to Sentence Length m OVERTRAINING Problem

5 CS730B Statistical NLP 2. Transformation-Based Error-Driven Learning o Phrase Structure Learning Algorithm m Initial State Naively Annotating Text. POS Tagging : Most Likely Tag. PP-Attachment : Low. Word Classification : Nouns. m Learning State Comparison to the Truth. : manually annotated Corpus. m Making the Transformation : RULE Added to the list of transformation. Sentences tagged with parts of Speech and returning a B-tree Structure with Nonterminals unlabelled. UnannotatedText InitialState Annotated Text LearningPS Rules [ Truth ] CorpusData

6 CS730B Statistical NLP 3. Learning Phrase Structure o Initial State of parser m Right branching parenthesis. m Final punctuation is attached high. m [Ex] : (( The ( dog ( and ( old ( cat ate ) ) ) ) ). ) o Structural Transformations m Transformation Type (1-8) ; (Add/Delete) a (Left/Right) parenthesis to the (Left/Right) of POS Tag “X” (9-12) ; (Add/Delete) a (Left/Right) parenthesis between tags X and Y m Example :: (( The ( dog barked ) ). ) Delete a left parenthesis to the right of “X” Add a right parenthesis to the right of “YY” Add a right parenthesis to the right of “Noun”

7 CS730B Statistical NLP 3.1. Examples o “Delete the left parenthesis to the right of “determiner” m Inits0 ( ( The ( dog barked ) ). ) m (step1) Delete the left paren to the right of deternminer ( ( The # dog barked ) ). ) m (step2) Delete the right paren that matches the just deleted paren ( ( The dog barked # ). ) m (step3) Add a left paren to the left of the constituent immediately to the left of the deleted left paren of the deleted left paren ( ( ( The dog barked ). ) m (step4) Add a right paren to the right of the constituent immediately to the right of the deleted left paren right of the deleted left paren ( ( ( The dog ) barked ) ). ) m If there is no constituent immediately to the right, or none immediately to the left, then the transformation fails to apply (redundancy) then the transformation fails to apply (redundancy)

8 CS730B Statistical NLP 3.2. Learning Transformation o Process Initialization Initialization with naïve parser 12 transformation templates Applying the 12 transformation templates to the sentence Best Transformation ( 가장 많은 변화를 주는 일반적인 “ 변형 ” 을 찾는다.) Best Transformation is found for the structures output by the parser in its current state ( 가장 많은 변화를 주는 일반적인 “ 변형 ” 을 찾는다.) Transformation Transformation is applied to the output resulting from bracketing the corpus using the parser in its current state Ordered list of transformation Transformation is added to the end of the Ordered list of transformation Looping Looping until no transformation found

9 CS730B Statistical NLP (continued) o Learning Transformation Application m Parsing the fresh text Naïve parsing  List of best scored transformation applied o Measure of Success :: Percentage of Constituent (PoC) m comparison to the correct PS description of training corpus. m from sentences output by our system which do not cross any constituents in the Penn Treebank structural description of the sentence. m ( ( ( The big ) ( dog ate ) ). )  ( ( ( The big dog ) ate ). )  PoC = 2/4 o Example m Best Scored “7” Transformation in WSJ Corpus Mostly “Noun phrases extraction” Transformation ( ( The ( cat meowed ) ). )  ( ( The cat ) meowed ). ) ( ( We ( ran (, ( and (they walked ) ) ) ). )  ( ( We ran ) (, ( and (they walked ) ) ) ). )

10 CS730B Statistical NLP 4. Results o ATIS corpus (Test Corpus 1) m training corpus = 21% size / Sentence Length = 11.3 (words)  “p222 와 비교 ” No crossing constituents = 60% Fewer than two crossing constituents = 74% Fewer than three crossing constituents = 85% m (Fig2) Percentage correct as a function of the number of transformations OVERTRAINING by specifically learned TR = small percent TS Solution = Set the Threshold :: specify the min level of improvements

11 CS730B Statistical NLP ( Continued ) m Random binary branching structure initialization drop the initial right-linear assumption with final punctuation high Total 147 Transformation and 87.13% bracketing accuracy o WSJ corpus (More complex corpus) m Table 2, Table 3, Table 4 Inside-Outside Algorithm 90.2% in “1095” 1-15-Sentence (11.3word) Sentence Length Bandwidths  Number of Transformation , Bracketing accuracy . Training Corpus Size  Number of Transformation , Bracketing accuracy . m Random binary branching structure initialization ( “250” 2-15-Sentence ) Total 325 Transformation and 84.72% bracketing accuracy m Sentence Length distribution Figure. 3

12 CS730B Statistical NLP 5. Conclusion o New Approach to learning a grammar to automatically parse text  “Transformation Template & Induced Rule” o The result is relatively high accuracy and effective (weakly statistical) statistical) o Next Project : Automatically Non-terminal labeling Algorithm o Advanced Transformation Procedure Experiments