LING 581: Advanced Computational Linguistics Lecture Notes February 19th.

Slides:



Advertisements
Similar presentations
School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING Chunking: Shallow Parsing Eric Atwell, Language Research Group.
Advertisements

LING 581: Advanced Computational Linguistics Lecture Notes February 2nd.
Statistical Methods Lynne Stokes Department of Statistical Science Lecture 7: Introduction to SAS Programming Language.
Albert Gatt Corpora and Statistical Methods Lecture 11.
LING 388: Language and Computers Sandiway Fong Lecture 2.
Chunk Parsing CS1573: AI Application Development, Spring 2003 (modified from Steven Bird’s notes)
LING 581: Advanced Computational Linguistics Lecture Notes February 9th.
Probabilistic Parsing Chapter 14, Part 2 This slide set was adapted from J. Martin, R. Mihalcea, Rebecca Hwa, and Ray Mooney.
In Search of a More Probable Parse: Experiments with DOP* and the Penn Chinese Treebank Aaron Meyers Linguistics 490 Winter 2009.
LING 581: Advanced Computational Linguistics Lecture Notes January 19th.
Shallow Parsing CS 4705 Julia Hirschberg 1. Shallow or Partial Parsing Sometimes we don’t need a complete parse tree –Information extraction –Question.
LING 581: Advanced Computational Linguistics Lecture Notes February 2nd.
LING 581: Advanced Computational Linguistics Lecture Notes February 23rd.
LING 581: Advanced Computational Linguistics Lecture Notes January 26th.
LING 581: Advanced Computational Linguistics Lecture Notes February 16th.
PCFG Parsing, Evaluation, & Improvements Ling 571 Deep Processing Techniques for NLP January 24, 2011.
LING 581: Advanced Computational Linguistics Lecture Notes January 26th.
Partial Prebracketing to Improve Parser Performance John Judge NCLT Seminar Series 7 th December 2005.
1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 22, 2004.
LING 581: Advanced Computational Linguistics Lecture Notes January 19th.
Probabilistic Parsing Ling 571 Fei Xia Week 5: 10/25-10/27/05.
SI485i : NLP Set 9 Advanced PCFGs Some slides from Chris Manning.
Tree Kernels for Parsing: (Collins & Duffy, 2001) Advanced Statistical Methods in NLP Ling 572 February 28, 2012.
BİL711 Natural Language Processing1 Statistical Parse Disambiguation Problem: –How do we disambiguate among a set of parses of a given sentence? –We want.
Probabilistic Parsing Reading: Chap 14, Jurafsky & Martin This slide set was adapted from J. Martin, U. Colorado Instructor: Paul Tarau, based on Rada.
GALE Banks 11/9/06 1 Parsing Arabic: Key Aspects of Treebank Annotation Seth Kulick Ryan Gabbard Mitch Marcus.
LING 581: Advanced Computational Linguistics Lecture Notes February 12th.
1 Statistical Parsing Chapter 14 October 2012 Lecture #9.
CSA2050: Introduction to Computational Linguistics Part of Speech (POS) Tagging II Transformation Based Tagging Brill (1995)
12/06/1999 JHU CS /Jan Hajic 1 Introduction to Natural Language Processing ( ) Statistical Parsing Dr. Jan Hajič CS Dept., Johns Hopkins Univ.
LING 581: Advanced Computational Linguistics Lecture Notes February 16th.
AQUAINT Workshop – June 2003 Improved Semantic Role Parsing Kadri Hacioglu, Sameer Pradhan, Valerie Krugler, Steven Bethard, Ashley Thornton, Wayne Ward,
Semi-supervised Training of Statistical Parsers CMSC Natural Language Processing January 26, 2006.
11 Chapter 14 Part 1 Statistical Parsing Based on slides by Ray Mooney.
Transformation-Based Learning Advanced Statistical Methods in NLP Ling 572 March 1, 2012.
Conversion of Penn Treebank Data to Text. Penn TreeBank Project “A Bank of Linguistic Trees” (as of 11/1992) University of Pennsylvania, LINC Laboratory.
Dependency Parser for Swedish Project for EDA171 by Jonas Pålsson Marcus Stamborg.
CSA2050 Introduction to Computational Linguistics Parsing I.
Supertagging CMSC Natural Language Processing January 31, 2006.
LING/C SC 581: Advanced Computational Linguistics Lecture Notes Feb 5 th.
1 Semi-Supervised Approaches for Learning to Parse Natural Languages Slides are from Rebecca Hwa, Ray Mooney.
Statistical Machine Translation Part II: Word Alignments and EM Alex Fraser Institute for Natural Language Processing University of Stuttgart
CSA2050: Introduction to Computational Linguistics Part of Speech (POS) Tagging II Transformation Based Tagging Brill (1995)
Integrating linguistic knowledge in passage retrieval for question answering J¨org Tiedemann Alfa Informatica, University of Groningen HLT/EMNLP 2005.
LING/C SC 581: Advanced Computational Linguistics Lecture Notes Feb 3 rd.
CIS Treebanks, Trees, Querying, QC, etc. Seth Kulick Linguistic Data Consortium University of Pennsylvania
LING/C SC 581: Advanced Computational Linguistics Lecture Notes Feb 17 th.
LING 581: Advanced Computational Linguistics Lecture Notes February 24th.
LING 581: Advanced Computational Linguistics Lecture Notes March 9th.
LING 581: Advanced Computational Linguistics Lecture Notes March 2nd.
Natural Language Processing Vasile Rus
CSC 594 Topics in AI – Natural Language Processing
Probabilistic CKY Parser
LING/C SC/PSYC 438/538 Lecture 21 Sandiway Fong.
LING 581: Advanced Computational Linguistics
LING/C SC 581: Advanced Computational Linguistics
LING 581: Advanced Computational Linguistics
LING 581: Advanced Computational Linguistics
CSCI 5832 Natural Language Processing
LING 581: Advanced Computational Linguistics
LING/C SC 581: Advanced Computational Linguistics
Constraining Chart Parsing with Partial Tree Bracketing
LING/C SC 581: Advanced Computational Linguistics
Chunk Parsing CS1573: AI Application Development, Spring 2003
LING/C SC 581: Advanced Computational Linguistics
LING/C SC 581: Advanced Computational Linguistics
LING/C SC 581: Advanced Computational Linguistics
Graphing Using Processing
LING/C SC/PSYC 438/538 Lecture 3 Sandiway Fong.
LING/C SC 581: Advanced Computational Linguistics
Presentation transcript:

LING 581: Advanced Computational Linguistics Lecture Notes February 19th

Bikel Collins and EVALB How did you guys do with Bikel-Collins on section 23?

Steps Running evalb Need: – parameter file: COLLINS.prm – Parser output: test.txt.parsed – Gold file: test.gold

Steps (Assume my data files are on my Desktop and EVALB is installed in my research subdirectory) –../research/EVALB/evalb -p../research/EVALB/COLLINS.prm test.gold test.txt.parsed Recall 83.8, Precision 89.2

Task from last time Run section 23 of the WSJ Treebank on the Bikel Collins parser – Extract the sentences from section 23 (perl etc.) Then run EVALB on the section to see how the Bikel Collins parser scores. – C code: (OSX) comment out the malloc line Report back next time.

System Flowchart Diagram: Bikel Collins parser Train Bikel Collins parser Train Treebank trees.mrg Events.obj.gz create using cat Bikel Collins parser Parse Bikel Collins parser Parse Treebank sentences.txt Parse trees.txt.parsed (one tree per line) tregex tsurgeon (-s flag) tregex View tregex View WSJ treebank 00–24 WSJ treebank 00–24 Sec 23 trees.mrg Sec 23 gold trees.mrg (one tree per line) EVALB recall precision F-measure ≈86% create using cat How? tregex View Search tregex View Search COLLINS.prm

Getting EVALB running Download from NYU – written in (plain) C Makefile: 1.all: evalb 2.evalb: evalb.c 3. gcc -Wall -g -o evalb evalb.c OSX: – make – gcc -Wall -g -o evalb evalb.c – evalb.c:25:20: fatal error: malloc.h: No such file or directory – compilation terminated..h: header file contains function stubs memory allocation malloc() defined in the C standard library stdlib.h target dependency command

Getting EVALB running Google the error and osx Use Spotlight – (3 copies) – /usr/include/malloc/malloc.h – /usr/include/sys/malloc.h It should compile now… (but with many warnings about type mismatches) or simply remove or comment out line 25

Extracting the sentences We need to extract the sentences to run the Bikel Collins parser Section 23 contains 2416 sentences There are many possible ways to do this: let's look at a couple …

Extracting the sentences Use the sec 23 gold trees: Perl regex: Exclude -NONE-

Extracting the sentences Use the POS tagged data in TREEBANK_3:

Extracting the sentences Use the POS tagged data in TREEBANK_3: Careful with blank lines

Extracting the sentences Use tregex Save matched sentences:

Extracting the sentences Use tregex Save Sentences: Perl regex \*T\*-[0-9]+ \*-[0-9]+ \*U\* \b0\b \b\*\b

Summary – WSJ corpus: sections 00 through 24 – Training: normally (20 sections) concatenate mrg files as input to Bikel Collins trainer training is fast – Evaluation: on section 23 use tsurgeon to get data ready for EVALB Use either the.pos files or tregex to grab the sentences for parsing sentences in section 23, parsing is slow… Can split the sentences up and run on multiple processors

New Homework Question: – How much training data do you need? Homework: – How does Bikel Collins vary in precision and recall on test data if you randomly pick out of 25 sections to do the training with? Test section: – I want you to pick a test section that’s not section 23 Use EVALB – plot precision and recall graphs

Summary Question: – How much training data do you need? Homework: – How does Bikel Collins vary in precision and recall on test data if you randomly pick out of 25 sections to do the training with? Test section: – I want you to pick a test section that’s not section 23 Use EVALB – plot precision and recall graphs

Perl Code Generate training data function shuffle

Training Data

What does the graph for the F-score (Labeled Recall/Labeled Precision) look like? F-score # of sentences used for training report your results next time…

Robustness and Sensitivity it’s often assumed that statistical models are less brittle than symbolic models 1. can get parses for ungrammatical data 2. are they sensitive to noise or small perturbations?

Robustness and Sensitivity Examples 1.Herman mixed the water with the milk 2.Herman mixed the milk with the water 3.Herman drank the water with the milk 4.Herman drank the milk with the water f(water)=117, f(milk)=21 (mix)(drink)

Robustness and Sensitivity Examples 1.Herman mixed the water with the milk 2.Herman mixed the milk with the water 3.Herman drank the water with the milk 4.Herman drank the milk with the water different PP attachment choices (low) (high) logprob = logprob = -47.2

Robustness and Sensitivity First thoughts... does milk forces low attachment? (high attachment for other nouns like water, toys, etc.) Is there something special about the lexical item milk? 24 sentences in the WSJ Penn Treebank with milk in it, 21 as a noun

Robustness and Sensitivity First thoughts... Is there something special about the lexical item milk? 24 sentences in the WSJ Penn Treebank with milk in it, 21 as a noun but just one sentence (#5212) with PP attachment for milk Could just one sentence out of 39,832 training examples affect the attachment options?

Robustness and Sensitivity Simple perturbation experiment – alter that one sentence and retrain parser sentences parses derived counts wsj obj.gz

✕ Robustness and Sensitivity Simple perturbation experiment – alter that one sentence and retrain delete the PP with 4% butterfat altogether

Robustness and Sensitivity Simple perturbation experiment – alter that one sentence and retrain or bump it up to the VP level Treebank sentences wsj mrg Derived counts wsj obj.gz training the Bikel/Collins parser can be retrained quicky

Robustness and Sensitivity Result: – high attachment for previous PP adjunct to milk Could just one sentence out of 39,832 training examples affect the attachment options? YES Why such extreme sensitivity to perturbation? logprobs are conditioned on many things; hence, lots of probabilities to estimate smoothing need every piece of data, even low frequency ones

Details… Two sets of files:

Details…

Robustness and Sensitivity (Bikel 2004): – “it may come as a surprise that the [parser] needs to access more than 219 million probabilities during the course of parsing the 1,917 sentences of Section 00 [of the PTB].''

Robustness and Sensitivity Trainer has a memory like a phone book:

Robustness and Sensitivity (mod ((with IN) (milk NN) PP (+START+) ((+START+ +START+)) NP-A NPB () false right) 1.0) – modHeadWord (with IN) – headWord (milk NN) – modifier PP – previousMods (+START+) – previousWords ((+START+ +START+)) – parent NP-A – head NPB – subcat () – verbIntervening false – side right (mod ((+STOP+ +STOP+) (milk NN) +STOP+ (PP) ((with IN)) NP-A NPB () false right) 1.0) – modHeadWord (+STOP+ +STOP+) – headWord (milk NN) – modifier +STOP+ – previousMods (PP) – previousWords ((with IN)) – parent NP-A – head NPB – subcat () – verbIntervening false – side right Frequency 1 observed data for: (NP (NP (DT a)(NN milk))(PP (IN with)(NP (ADJP (CD 4)(NN %))(NN butterfat))))

Robustness and Sensitivity 76.8% singular events 94.2% 5 or fewer occurrences

Robustness and Sensitivity Full story more complicated than described here... by picking different combinations of verbs and nouns, you can get a range of behaviors VerbNounAttachment milk + nounnoun + milk drankwaterhigh mixedwaterlowhigh mixedcomputerlow f(drank)=0 might as well have picked flubbed