A COMPARISON OF HAND-CRAFTED SEMANTIC GRAMMARS VERSUS STATISTICAL NATURAL LANGUAGE PARSING IN DOMAIN-SPECIFIC VOICE TRANSCRIPTION Curry Guinn Dave Crist Haley Werth

Outline l Probabilistic language models »N-grams l The EPA project l Experiments

Probabilistic Language Processing: What is it? l Assume a note is given to a bank teller, which the teller reads as I have a gub. (cf. Woody Allen) l NLP to the rescue …. »gub is not a word »gun, gum, Gus, and gull are words, but gun has a higher probability in the context of a bank

Real Word Spelling Errors l They are leaving in about fifteen minuets to go to her house. l The study was conducted mainly be John Black. l Hopefully, all with continue smoothly in my absence. l Can they lave him my messages? l I need to notified the bank of…. l He is trying to fine out.

Letter-based Language Models l Shannon’s Game l Guess the next letter: l

Letter-based Language Models l Shannon’s Game l Guess the next letter: l W

Letter-based Language Models l Shannon’s Game l Guess the next letter: l Wh

l Shannon’s Game l Guess the next letter: l Wha Letter-based Language Models

l Shannon’s Game l Guess the next letter: l What Letter-based Language Models

l Shannon’s Game l Guess the next letter: l What d Letter-based Language Models

l Shannon’s Game l Guess the next letter: l What do Letter-based Language Models

l Shannon’s Game l Guess the next letter: l What do you think the next letter is? Letter-based Language Models

l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l Letter-based Language Models

l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What Letter-based Language Models

l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What do Letter-based Language Models

l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What do you Letter-based Language Models

l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What do you think Letter-based Language Models

l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What do you think the Letter-based Language Models

l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What do you think the next Letter-based Language Models

l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What do you think the next word is? Letter-based Language Models

Word-based Language Models l A model that enables one to compute the probability, or likelihood, of a sentence S, P(S). lSimple: Every word follows every other word w/ equal probability (0-gram) »Assume |V| is the size of the vocabulary V »Likelihood of sentence S of length n is = 1/|V| × 1/|V| … × 1/|V| »If English has 100,000 words, probability of each next word is 1/ =.00001

Word Prediction: Simple vs. Smart Smarter: probability of each next word is related to word frequency (unigram) – Likelihood of sentence S = P(w 1 ) × P(w 2 ) × … × P(w n ) – Assumes probability of each word is independent of probabilities of other words. Even smarter: Look at probability given previous words (N-gram) – Likelihood of sentence S = P(w 1 ) × P(w 2 |w 1 ) × … × P(w n |w n-1 ) – Assumes probability of each word is dependent on probabilities of other words.

Training and Testing l Probabilities come from a training corpus, which is used to design the model. »Overly narrow corpus: probabilities don't generalize »Overly general corpus: probabilities don't reflect task or domain l A separate test corpus is used to evaluate the model, typically using standard metrics »Held out test set

Simple N-Grams l An N-gram model uses the previous N-1 words to predict the next one: »P(w n | w n-N+1 w n-N+2… w n-1 ) l unigrams: P(dog) l bigrams: P(dog | big) l trigrams: P(dog | the big) l quadrigrams: P(dog | chasing the big)

The EPA task l Detailed diary of a single individual’s daily activity and location l Methods of collecting the data: »External Observer »Camera »Self-reporting –Paper diary –Handheld menu-driven diary –Spoken diary

Spoken Diary l From an utterance like “I am in the kitchen cooking spaghetti”, map that utterance into »Activity(cooking) »Location(kitchen) l Text abstraction l Technique »Build a grammar »Example

Sample Semantic Grammar ACTIVITY_LOCATION -> ACTIVITY' LOCATION' : CHAD(ACTIVITY',LOCATION'). ACTIVITY_LOCATION -> LOCATION' ACTIVITY' : CHAD(ACTIVITY',LOCATION'). ACTIVITY_LOCATION -> ACTIVITY' : CHAD(ACTIVITY', null). ACTIVITY_LOCATION -> LOCATION' : CHAD(null,LOCATION'). LOCATION -> IAM LOCx' : LOCx'. LOCATION -> LOCx' : LOCx'. IAM -> IAM1. IAM -> IAM1 just. IAM -> IAM1 going to. IAM -> IAM1 getting ready to. IAM -> IAM1 still. LOC2 -> HOUSE_LOC' : HOUSE_LOC'. LOC2 -> OUTSIDE_LOC' : OUTSIDE_LOC'. LOC2 -> WORK_LOC' : WORK_LOC'. LOC2 -> OTHER_LOC' : OTHER_LOC'. HOUSE_LOC -> kitchen : kitchen_code. HOUSE_LOC -> bedroom : bedroom_code. HOUSE_LOC -> living room : living_room_code. HOUSE_LOC -> house : house_code. HOUSE_LOC -> garage : garage_code. HOUSE_LOC -> home : house_code. HOUSE_LOC -> bathroom : bathroom_code. HOUSE_LOC -> den : den_code. HOUSE_LOC -> dining room : dining_room_code. HOUSE_LOC -> basement : basement_code. HOUSE_LOC -> attic : attic_code. OUTSIDE_LOC -> yard : yard_code.

Statistical Natural Language Parsing l Use unigram, bigram and trigram probabilities l Use Bayes’ rule to obtain these probabilities: P(A|B) = P(B|A) * P(A)/ P(B) l The formula P(“kitchen”|30121 Kitchen) is computed by determining the percentage of times the word “kitchen” appears in diary entries that have been transcribed in the category Kitchen. l P(30121 Kitchen) is the probability that a diary entry is of the semantic category Kitchen. l P(“kitchen”) is the probability that “kitchen” appears in any diary entry. l Bayes’ rule can be extended to take into account each word in the input string.

The Experiment l Digital Voice Recorder + Heart Rate Monitor »Heart rate monitor will beep if the rate changes by more than 15 beats per minute between measurements (every 2 minutes)

Subjects IDSexOccupationAgeEducation 1F Manages Internet Company 52Some College 2FGrocery Deli Worker18Some College 3MConstruction Worker35High School 4FDatabase Coordinator29Graduate Degree 5FCoordinator for Non-profit56Some College 6MUnemployed50High School 7MRetired76High School 8MDisabled62High School 9M Environment Technician 56Graduate Degree

Recordings Per Day

Heart Rate Change Indicator Tones and Subject Compliance SNumber of Tones Per Day (Avg.) % of Times Subject Made a Diary Entry Corresponding to a Tone % % % % % % % % %

Per Word Speech Recognition PPer Word Recognition Rate (%)

Semantic Grammar Location/Activity Encoding Precision and Recall Word Rec. Rate LocationActivity PrecisionRecallPrecisionRecall Av

Word Recognition Accuracy’s Effect on Semantic Grammar Precision and Recall

Statistical Processing Accuracy Activity Accuracy Location Accuracy Hand- transcribed 86.7%87.5% Using speech Recognition 48.3%49.0%

Word Recognition Affects Statistical Semantic Categorization Rec. Rate % LocationActivity Accuracy % Av

Per Word Recognition Rate Versus Statistical Semantic Encoding Accuracy

Time, Activity, Location, Exertion Data Gathering Platform

Research Topics l Currently, guesses for the current activity and location are computed independently of each other »They are not independent! l Currently, guesses are based on the current utterance. »However, the current activity/location is not independent from previous activity/locations. l How do we fuse data from other sources (gps, beacons, heart rate monitor, etc.)?