Prosody and NLP Seminar by Nikhil: Adith: Prachur: 06D05011 We have a presentation this Friday ?

Abstract Speech Processing and Natural Language Processing share a common area of study: Language. However, over time, they have grown to have little in common regarding theoretical models or methods of analysis. NLP takes written text as the starting point for it's analysis, however, a lot of valuable information is lost in encoding speech as merely text. It is commonly accepted that intonational features of spoken language can greatly aid NLP tasks (like adjective scope resolution). We explore the foundations of the study of Prosody and observe some approaches that use prosodic cues to aid NLP.

Motivation Language is not text driven but speech driven. NLP currently has written text as the starting point for it's analysis (primarily due to the abundance of such data). A lot of information is lost on ignoring spoken features and just looking at the text.

A Way Out ? Utilize spoken features for NLP tasks NLP needs all the help it can get.  Dealing at the pragmatics or discourse level is extremely untenable  Prosodic cues carry useful pragmatic information

What is Prosody, exactly? Comes from Poetry, prosody refers to the study of poetic meters[1] (rhythms)‏ Written text treats words as the basic building blocks of language. Spoken language treats syllables as the basic building block.

Wikipedia has this to say : Prosody is the rhythm, stress, and intonation of connected speech (as opposed to smaller elements like syllables or words). Prosody may reflect :  features of the speaker,  emotional state of a speaker,  features of the utterance,  ironic or sarcastic,  emphasis, contrast, and focus What is Prosody, exactly? (contd.)

Intonation ? Conveys paralinguistic information, emphasis and contrast. Intonation on a particular word could differentiate between sentence moods. –You are finISHED (interrogative)‏ –You are FINIshed (imperative)‏ Image courtesy Google Image Search

Stress is applied on Content Words in spoken utterances. cOntent - Noun. "I really liked their presentation's content.“ contEnt - Verb. "I have done my best. I am content.“ Stress on a pair of words distinguishes between the syntactic role played by each word in the pair. tight rope : A rope that is held taut. tight-rope : A circus-act uses this contraption :)‏ [2] And Stress !

Courtesy tom The Dancing Bug

Prosodic cues Prosodic functions important for linguistics are –Marking of boundaries (syntactic, semantic or dialogue units.)‏ –Relative duration of phonetic segments –At syllable level : Energy, intensity, duration and intonation of syllable. We shall see two approaches of using these features in tasks central to NLP.

Cue Used : Relative duration of phonetic segments Aim : To improve the parsing of ambiguous sentences. Method : Augmenting the syntax grammar with a few non-terminals and rules. Prosody-Augmented Syntax Grammars[3]

Concept of “Word Break Indices” used to show prosodic decoupling between neighboring words. E.g. - Andrea 1 moved 1 the 0 bottle 3 under 0 the 0 bridge. Andrea 1 moved 3 the 0 bottle 1 under 0 the 0 bridge. Break indices were generated by analysing the coda that have a pause. Coda is the final consonant of a word E.g. – cup, milk

Grammar Modification Original grammar rules like S -> NP VP etc. are changed to S -> NP link1 VP. “Link” non-terminals are used for the word- break indices. –For rules like NP ->  we allow rules of the form link -> . –To prevent spurious parses due to the introduction of empty links, we need some constraints which can be easily incorporated

Results The incorporation of prosody resulted in a reduction of about 25% in the number of parses found. Parse times increase about 37%. Extremely common cases of syntactic ambiguity can be resolved with prosodic information, and that grammars can be modified to take advantage of prosodic information for improved parsing

Using Prosodic Features in Language Models[4] The outlined approach uses syllable-based prosodic cues, namely –Duration of the syllable –Average energy (intensity)‏ –The average F0 (fundamental frequency of the syllable) contour –The slope of the F0 contour (visualised as intonation-rising or falling/flat)‏

Recognition of Prosodic Features

Prosody in Language Model We want to measure P(w n | w n-1,w n-2,…,F) Naively modelled by linear interpolation : –Assumption : prosody features independent of previous words (not true!!). P(w n | w n-1,w n-2,…,F) = αP(w n | w n-1,w n-2,…) + (1- α)P(w n |F) We want something better

Factored Language Model Instead of a word W we will deal with a set of word-factors F={f1,f2..fk} (Factors may include the word itself)

Here, F is chosen as {W, prosodic features} The four prosodic features are encoded as binary numbers(s0 to s15). These numbers are assigned to each syllable of the word. For e.g. the prosodic representation for the word “Actually” can be either ‘s10s12s6’ or ‘s10s15s6’.

Conclusion Prosodic Cues can play an important role as a heuristic for many NLP tasks. All is not one way traffic though. POS tagging (since its relatively accurate) is used to aid speech synthesis tasks which conventionally used only prosodic cues[5]

Future Work Handling prosodic information is a first step towards integration of Speech Processing and NLP Courtesy ZITS

References 1.Wikipedia 2.Fromkin, Rodman and Hyams, An Introduction to Language, 7 th Ed, Thomson and Wadsworth 3.John Bear and Patti Price (1990), “Prosody, Syntax and Parsing”, in proceedings of the 28 th annual meeting of the ACL 4.Songfang Huang and Steve Renals (2007), “Using Prosodic Features in Language Models for Meetings”, IRTG annual meeting 5. rts.pdf