Automatic Morphology and Minimum Description Length John Goldsmith Department of Linguistics

Today’s plan 1 A computer program -- what it looks like, what it does. 2 The framework -- Minimum Description Length 3 Situate MDL within a linguistic context... Comparison with Early Generative Grammar 4 Situate MDL within a broader intellectual context 5 More substantive description of Automorphology’s design 6 The broader perspective

Today’s plan 1 A computer program -- what it looks like, what it does. 2 The framework -- Minimum Description Length 4 Comparison with Early Generative Grammar 3 Situate MDL within a linguistic context... 5 Situate MDL within a broader intellectual context 6 More substantive description of Automorphology’s design 7 Consequences for anthropology

WinAutomorphology 1 n A version available on the web at goldsmith n A C++ Windows program that accepts data as input and provides a morphological analysis....

Automorphology

n What do you have to put into a program like that? How much do you have to put into a program like that? n That is, does it have to have a lot of innate knowledge? Does it help for it to have a lot of innate knowledge? n If you build such a program, how do you know if it does it the same way as a child?

What do we want? If you give the program a computer file containing Tom Sawyer, it should tell you that the language has a category of words that take the suffixes ing,s,ed, and NULL; another category that takes the suffixes 's, s, and NULL; If you give it Jules Verne, it tells you there's a category with suffixes: a aient ait ant (chanta, chantaient, chantait, chantant)

n And it should tell you about irregular stem allomorphy if your language contains it.

That's what AutoMorphology does. How much data do you need? n You get reasonable results fast, with 5,000 words, but results are much better with 50,000, and much better with 500,000 words (length of corpus).

Unsupervised learning... n No prepared corpus; no tagging; just the facts. n The goal is to reconstruct the logic of linguistics in a quantitative fashion (to the extent that is necessary).

Unsupervised learning n A fully explicit linguistic hypothesis. n A device (an algorithm) with immediate practical uses. n Arguably the embodiment of linguistic theory: the explicit and quantifiable specification of the relationship between data and analysis (grammar).

n For the purposes of version 1 of AutoMorphology, I will restrict myself to Indo-European languages, and in general languages in which the average number of suffixes per word is not greater than 2. (We drop this requirement in AutoMorphology 2.)

Today’s plan 1 A computer program -- what it looks like, what it does. 2 The framework -- Minimum Description Length 3 Situate MDL within a linguistic context... Comparison with Early Generative Grammar 4 Situate MDL within a broader intellectual context 5 More substantive description of Automorphology’s design 6 The broader perspective

Minimum Description Length Jorma Rissanen (1989) Data Analyzer Analysis Select the analyzer and analysis such that the sum of their lengths is a minimum.

Data Analyzer Analysis Analyzer Analysis Analyzer Analysis Analyzer Analysis Analyzer Analysis Etc...

The challenge Is to find a means of quantifying n the length of an analyzer, and n the length of an analysis

“Compressed form of data?” Think of data as a dense, rich, detailed description (evidence), and Think of compressed form as n Description in high level language + n Description of the particulars of the event in question (a.k.a. boundary conditions, etc.)...

“Analyzer” Is the set of statements that allows translation between high-level and low- level descriptions.

Minimizing sum of length of Analyzer + Compressed form of data = Aim for conciseness in high-level description + Principles of analysis

Don’t overlook the fact... …that the goal of MDL analysis is nothing less than the solution of the problem of induction. How do we justify generalization, given evidence?

the problem of induction Speechchild/linguistic theory grammar Datascientisttheory Sensebrainthought/percept Evidencemindbelief

Today’s plan 1 A computer program -- what it looks like, what it does. 2 The framework -- Minimum Description Length 4 Comparison with Early Generative Grammar 3 Situate MDL within a linguistic context... 5 Situate MDL within a broader intellectual context 6 More substantive description of Automorphology’s design 7 Consequences for anthropology

Data Morphological analyser Morphological analysis of that corpus “signature”

Very simply put... n Just state “ed” “s” “ing” “heit” “ité” once in the grammar; n pay for its occurrence (how many bits does it take to pay for those few letters) just once; n then make repeated reference (use pointers) to those entries.

References, pointers... n Are not free. n Information theory tells us exactly what they cost. The fundamental measure is Shannon’s: a pointer to an item of reference frequency P out of a universe of N possibilties is of length: log (N/P)

Summing over all items, and weighting by count gives us the famous formula:

A probabilistic morphology: n Assigns a probability to all words that it can generate; and these probabilities must add up to 1.0. n A word is three choices: –choice of signature –choice of stem within signature –choice of suffix within signature

n Each of those is assigned a probability, based on counts. n Probability of a signature

Similarly, the probability of a stem is the number of times of its occurrence divided by the number of occurrences of that signature in the corpus.

Likewise for the suffixes… If the analysis is wrong, the numbers will be much worse than if it’s right. “The numbers” a model of frequencies of words.

Maximum Likelihood n The best morphology is the one that assigns the highest probability to the observed data. n …known in the biz as Maximum Likelihood.

Today’s plan 1 A computer program -- what it looks like, what it does. 2 The framework -- Minimum Description Length 4 Comparison with Early Generative Grammar 3 Situate MDL within a linguistic context... 5 Situate MDL within a broader intellectual context 6 More substantive description of Automorphology’s design 7 Consequences for anthropology

Compare with Early Generative Grammar (EGG) Data Linguistic Theory Analysis 1 Analysis 2 Preference: A1/A2

Linguistic theory Data Analysis Linguistic theory Data Analysis Yes/No Linguistic theory Analysis 1 Analysis 2 Data 1 is better/ 2 is better

Implicit in EGG was the notion... that the best Linguistic Theory could be selected by... Getting a set of n candidate LTs; submitting to each a set of corpora; search (using unknown heuristics) for best analyses of each corpus within each LT; The LT wins for whom the sum total of all of the analyses is the smallest.

No cost to UG n In EGG, there was no cost associated with the size of UG -- in effect, no plausibility measure.

In MDL, in contrast…. n we can argue for a grammar for a given corpus. n We can also argue at the Linguistic Theory level if we so choose...

n Select n corpora, and select that LT on the basis of LT’s length plus the length of all of the grammars derived from it, plus the lengths of the compressed corpora derived from those grammars. n Pick the LT with the shorted some total length.

Today’s plan 1 A computer program -- what it looks like, what it does. 2 The framework -- Minimum Description Length 4 Comparison with Early Generative Grammar 3 Situate MDL within a linguistic context... 5 Situate MDL within a broader intellectual context 6 More substantive description of Automorphology’s design 7 Consequences for anthropology

Today’s plan 1 A computer program -- what it looks like, what it does. 2 The framework -- Minimum Description Length 4 Comparison with Early Generative Grammar 3 Situate MDL within a linguistic context... 5 Situate MDL within a broader intellectual context 6 More substantive description of Automorphology’s design 7 Consequences for anthropology

Distinction between heuristics and “theory” n In the context of MDL, the heuristics are extratheoretical, but from the point of view of the (psycho-)linguist, they are very important. n The heuristics propose; the theory disposes.

Stems with their signatures abrupt NULL ly ness. abs ence ent. absent -minded NULL ia ly. absent-minded NULL ly absentee NULL ism absolu NULL e ment. absorb ait ant e er é ée abus ait er abîm e es ée.

Now build up signature collection... Top 10, 100K words 1.NULL.ed.ing NULL.ed.ing.s NULL.s 's.NULL.s NULL.ed.s NULL.ly NULL.ed 's.NULL NULL.d.s NULL.ing

Verbose signature....NULL.ed.ing. 58 heapcheckrevolt plunderlookobtain escortproclaimarrest gaindestroystay suspectkillconsent knocktracksucceed answerfrightenglitter....

Stem allomorphy In a corpus of French, we find pairs of stems: ç:c/_# 10commenç\commenc menaç\menac renonç\renonc avanç\avanc annonç\annonc s'effaç\s'effac enfonç\enfonc recommenç\recommenc perç\perc forç\forc lanç\lanc

compressed length of corpus:

Heuristics Find more than one stem that commutes with more than one suffix

n Negotiate for where the stem/suffix break should be: mea all take the suffixes n/ns. christia roma reig rui saxo tow

Today’s plan 1 A computer program -- what it looks like, what it does. 2 The framework -- Minimum Description Length 4 Comparison with Early Generative Grammar 3 Situate MDL within a linguistic context... 5 Situate MDL within a broader intellectual context 6 More substantive description of Automorphology’s design 7 Consequences for the broader context

Data Mind/head/brain/ nervous system Analysis But are the contributions of these two of equal magnitude, in the case of language? Otherwise put, to what extent does the structure here reside in the data -- and to what extent in the analyzer?

A rich, deductive structure? n No shadow of a rich, deductive structure in the learner casting its image on the form of the learned morphology. n A pure structuralism -- a structuralism without Jakobsonian dualities (but see my paper on Jakobson….)