Morphological Segmentation Inside-Out Ryan Cotterell, Arun Kumar, Hinrich Schütze

Old Idea: Surface Morphological Segmentation We are going to give examples in English, but other languages are far more complex!

unachievability un achiev abil ity Segment PREFIX STEM SUFFIX SUFFIX One common way of processing morphology is what we are going to call *surface* morphological segmentation. The goal, roughly speaking, is to separate a surface form of a word into its sequence of morphemes. Perhaps with a labeling. This task has attracted a lot of attention over the years with a number supervise and unsupervised methods being proposed. PREFIX STEM SUFFIX SUFFIX

Semi-New Idea: Canonical Morphological Segmentation

unachievability unachieveableity un achieve able ity Restore Segment DEFINE UNDERLYING FORM This work focuses on a different formulation of the task: canonical segmentation. The goal here is to map the surface form into an underlying form and *then* segment it. To point out the differences, compared to the last slide, we have added an "e" to "achieve" and mapped "abil" to "able". un achieve able ity PREFIX STEM SUFFIX SUFFIX

Why is canonicalization useful? Here's why you should care about this problem. Segmenting words alone is not enough. We eventually need to reason about the relationships between words. When we perform canonical segmentation, it becomes immediately clear, which words share morphemes.

unachievability achievement underachiever achieves Segmentation does not happen in isolation. Ideally, we would like to analyze all the word's in a language's lexicon achieves

un achiev abil ity achieve ment under achiev er achieve s

Are they the same morpheme??? un achiev abil ity achieve ment under achiev er achieve s

unachievability achievement underachiever achieves Segmentation does not happen in isolation. Ideally, we would like to analyze all the word's in a language's lexicon achieves

unachieveableity achievement underachieveer achieves

un achieve able ity achieve ment under achieve er achieve s

Canonical segmentations are standardized across words un achieve able ity achieve ment under achieve er Better preprocessing, e.g., more meaningful reduction in sparsity and reasoning about compositionality achieve s

unachievability thinkable accessible untouchable Segmentation does not happen in isolation. Ideally, we would like to analyze all the word's in a language's lexicon untouchable

unachieveableity thinkable accessable untouchable Segmentation does not happen in isolation. Ideally, we would like to analyze all the word's in a language's lexicon untouchable

un achieve able ity think able access able un touch able

un achieve able ity think able access able un touch able

New Idea: Morphology as Parsing

unachievability achievement underachiever achieves Segmentation does not happen in isolation. Ideally, we would like to analyze all the word's in a language's lexicon achieves

unachieveableity achievement underachieveer achieves

un achieve able ity achieve ment under achieve er achieve s

un achieve able ity achieve ment under achieve er achieve s

under achieve er

under achieve er

under achieve er

under achieve er

PREFIX STEM SUFFIX under achieve er

Why are trees useful? Here's why you should care about this problem. Segmenting words alone is not enough. We eventually need to reason about the relationships between words. When we perform canonical segmentation, it becomes immediately clear, which words share morphemes.

Reason 1: Words are ambiguous! Tree Captures Ambiguity! SUFFIX PREFIX STEM SUFFIX PREFIX STEM un lock able un lock able “capable of being unlocked” “incapable of being locked” PREFIX STEM SUFFIX un lock able “???” Flat Segmentation Doesn’t!

Reason 2: Model Order of Affixation Path of Derivation achieve underachieve underachiever Encoded As Tree More Features PREFIX STEM SUFFIX under achieve er

New Resource To the best of our knowledge, the fully supervised version of this task has never been considered before in the literature so introduce a novel joint probability model.

Morphological Tree Bank English Size

A Joint Model To the best of our knowledge, the fully supervised version of this task has never been considered before in the literature so introduce a novel joint probability model.

Canonical Segmentation Parse Tree unachieveableity Underlying Form unachievability un achieve able ity Canonical Segmentation Parse Tree unachieveableity unachievability un achieve able ity We model the probability of a canonical segmentation – CLICK and an underlying form – CLICK given the surface form of a word – CLICK CLICKThe first factor scores a canonical segmentation underlying form pair. Basically, it asks how good is this pair? For example, un - achieve - able -ity and achieavility. This a structured factor and can be seem as the score of a semi-Markov model.CLICKThe second factror scores a surface segmentation, underlying form pair. Basically, it asks how good is this pair? Now, this notation belies a bit of the complexity. This factor is, again, structured. In fact, in general we have to encoder all possible alignmenet between the two strings. Luckily, we can encode this as a weighted finite-state machine. The paper explains this in detail.CLICKWe put them all together and we get our model. The remaining details such as the feature templates can be found in the paper.PAUSECLICK Word (Surface Form)

(s=un achieve able ity, u=unachieveableity) How good is the tree- underlying form pair? (s=un achieve able ity, u=unachieveableity) How good is the underlying form-word pair? We define this model as being proportional the exponential of a linear model. We can see this as being composed of two difference factors. (u=unachieveableity, w=unachievability)

Inference and Learning Inference is intractable! Approximate inference with importance sampling Decoding also with importance sampling Learning AdaGrad (Duchi et al. 2011) Unfortunately, marginal inference in our model is intractable! We explain why in the paper. As the model is globally normalized, even computing a gradient requires inference. To solve this, we rely on an approximation known as importance sampling. At a high-level, importance sampling takes smaples from an easy-distribution and lets the model rescore them. Decoding a.k.a. MAP infernece also intractable, but, again, we can approximately solve this with importance sampling.Once we get our approximate gradient, using importance sampling, we train the model with AdaGrad.CLICK

Experimental Results Key Point: Do trees help segmentation accuracy? Baseline: flat segmentation model New Task:

Results

Fin. Thank You!