1. Topics in Linguistics ENG 331
Rania Al-Sabbagh
Department of English
Faculty of Al-Alsun (Languages)
Ain Shams University
Week 10

2. Raw Corpora – Limited Benefits
Raw corpora are of limited use.
The best we can do with a raw corpus is to get a wordlist.
What do we use wordlists for?
Or do some complex searches using RegEx.
Give examples!
However, to make the most of a corpus, we may need to annotate it; that is to add linguistic information to the texts.
Linguistic information can be of any type: phonetic, phonological, morphological, syntactic etc.
Week 10

3. Automatic Corpus Annotation
Corpus annotation is a very tedious process.
Imagine how much time you need to manually each word for its grammatical class in a corpus of 1M tokens.
It is always good to look for automatic software to do the analysis. This is referred to automatic corpus annotation.
Some automatic software used for corpus annotation are:
Part-of-Speech (POS) taggers
Morphological analyzers
Syntactic parsers
Semantic role labelers
Week 10

4. Part of Speech Taggers 1
Part-of-Speech (POS) Taggers are computer software that automatically labels each word in context for its grammatical class.
In context means that the word must be within a phrase, a clause, or a sentence.
There are many POS taggers in the market, especially for English. Two of the widely used ones are:
CLAW
TreeTagger
Week 10

5. Part of Speech Taggers 2 Input: I have two dogs.
CLAWS Output: I_PNP have_VHB two_CRD dogs_SENT ._PUN
TreeTagger Output: I PP I
have VHP have
two CD two
dogs NNS dog
. SENT .
The differences you see are not only in the formatting (i.e. inline vs. stand-off), but also in the content (i.e. the tagset).
Week 10

6. Part of Speech Taggers 3
If there are many POS taggers out there, how can we select one?
Performance against a manually-annotated random sample.
The tagset and whether it gives me the information I
The tagset is the set of labels used to mark the grammatical class of each word.
Some POS taggers have a very concise tagset. For example, the CATiB POS tagger for Arabic has only 6 labels: NOM, VRB, VRB-PASS, PRT, PNX. If you are interested in studying prepositions, this tagset may not be good for you.
Week 10

7. Morphological Analyzers
Morphological analyzers – sometimes referred to as stemmers – split off affixes. One well known example for English is the Porter Stemmer.
Morphological analysis is specially important for Morphologically-Rich languages such as Arabic.
Can you guess why?
A well-known morphological analyzer for Arabic is MADAMIRA.
Week 10

8. Syntactic Parsers
Syntactic parsers are computer software that labels phrases for their syntactic types.
Examples of well-known syntactic parsers for English are:
LINK grammar
Stanford parser
ARK
Week 10

9. Semantic Role Labeling
Semantic role labeling analyzes the role of each word in the sentence as a subject, an object, an adjunct, etc.
One example of semantic role labelers for English is:
Week 10

10. Manual Corpus Annotation
Unfortunately, not all levels of linguistic analysis can be done automatically.
For example, to the best of my knowledge there is no software that can extract metaphors, or identify the type of speech act
Consequently, we have to go for the manual annotation option; where human beings are hired to read the corpus word for word and do the analysis we want them to do.
Although manual annotation is a very expensive and time consuming, it is typically more accurate than automatic annotation.
Week 10

11. Manual Annotation Modes
There are two modes for manual annotation:
In-lab: to hire expert human annotators and train them well and have them working under a controlled environment.
Crowdsourcing: to have a survey posted online and people from around the world doing the analysis for you. Two of the most famous platforms that host crowdsourcing annotation are Amazon Mechanical Turk and Crowdflower.
Do you think crowdsourcing works for all levels of language analysis?
What problems can we have with this mode of annotation?
Can we avoid them?
Week 10

12. In-Lab vs. Crowdsourcing
The pros and cons of each mode are as follows:
Crowdsourcing
In-Lab
Cheap
Expensive
Non-expert annotators
Expert annotators
Typically for simple tasks
Works for all tasks
Many scammers
Controlled environment
Week 10

13. Best Practices
For in-lab annotation, the researcher needs to prepare well-written annotation guidelines. All ad hoc decisions need to be written in the guidelines. The annotators should be trained on a sample corpus before working individually.
For crowdsourcing, the researcher can have an admission test at the beginning of the online survey, or repeat questions randomly throughout the survey.
How to handle disagreement?
For in-lab annotation, if there are 3+ annotators you can have a majority vote; otherwise, the researcher can settle disagreement him/herself or exclude the disagreement instances.
For crowdsourcing annotation, get as many survey takers as possible to get majority votes.
Week 12

14. Quiz True or False? Stemmers identify affixes.
All POS taggers use the same tagset.
Manual annotation uses computer software.
Part of speech taggers work at the word level.
Automatic annotation relies on human efforts.
Syntactic parsers analyze phrases and sentences.
Crowdsourcing is cheaper than in-lab annotation.
With crowdsourcing, you can select expert annotators.
Crowdsourcing might be less accurate than in-lab annotation.
Corpus annotation is adding linguistic information to the corpus texts.
Week 10