1. A COMPARISON OF HAND-CRAFTED SEMANTIC GRAMMARS VERSUS STATISTICAL NATURAL LANGUAGE PARSING IN DOMAIN-SPECIFIC VOICE TRANSCRIPTION Curry Guinn Dave Crist Haley Werth

2. Outline l Probabilistic language models »N-grams l The EPA project l Experiments

3. Probabilistic Language Processing: What is it? l Assume a note is given to a bank teller, which the teller reads as I have a gub. (cf. Woody Allen) l NLP to the rescue …. »gub is not a word »gun, gum, Gus, and gull are words, but gun has a higher probability in the context of a bank

4. Real Word Spelling Errors l They are leaving in about fifteen minuets to go to her house. l The study was conducted mainly be John Black. l Hopefully, all with continue smoothly in my absence. l Can they lave him my messages? l I need to notified the bank of…. l He is trying to fine out.

5. Letter-based Language Models l Shannon’s Game l Guess the next letter: l

6. Letter-based Language Models l Shannon’s Game l Guess the next letter: l W

7. Letter-based Language Models l Shannon’s Game l Guess the next letter: l Wh

8. l Shannon’s Game l Guess the next letter: l Wha Letter-based Language Models

9. l Shannon’s Game l Guess the next letter: l What Letter-based Language Models

10. l Shannon’s Game l Guess the next letter: l What d Letter-based Language Models

11. l Shannon’s Game l Guess the next letter: l What do Letter-based Language Models

12. l Shannon’s Game l Guess the next letter: l What do you think the next letter is? Letter-based Language Models

13. l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l Letter-based Language Models

14. l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What Letter-based Language Models

15. l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What do Letter-based Language Models

16. l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What do you Letter-based Language Models

17. l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What do you think Letter-based Language Models

18. l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What do you think the Letter-based Language Models

19. l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What do you think the next Letter-based Language Models

20. l Shannon’s Game l Guess the next letter: l What do you think the next letter is? l Guess the next word: l What do you think the next word is? Letter-based Language Models

21. Word-based Language Models l A model that enables one to compute the probability, or likelihood, of a sentence S, P(S). lSimple: Every word follows every other word w/ equal probability (0-gram) »Assume |V| is the size of the vocabulary V »Likelihood of sentence S of length n is = 1/|V| × 1/|V| … × 1/|V| »If English has 100,000 words, probability of each next word is 1/100000 =.00001

22. Word Prediction: Simple vs. Smart Smarter: probability of each next word is related to word frequency (unigram) – Likelihood of sentence S = P(w 1 ) × P(w 2 ) × … × P(w n ) – Assumes probability of each word is independent of probabilities of other words. Even smarter: Look at probability given previous words (N-gram) – Likelihood of sentence S = P(w 1 ) × P(w 2 |w 1 ) × … × P(w n |w n-1 ) – Assumes probability of each word is dependent on probabilities of other words.

23. Training and Testing l Probabilities come from a training corpus, which is used to design the model. »Overly narrow corpus: probabilities don't generalize »Overly general corpus: probabilities don't reflect task or domain l A separate test corpus is used to evaluate the model, typically using standard metrics »Held out test set

24. Simple N-Grams l An N-gram model uses the previous N-1 words to predict the next one: »P(w n | w n-N+1 w n-N+2… w n-1 ) l unigrams: P(dog) l bigrams: P(dog | big) l trigrams: P(dog | the big) l quadrigrams: P(dog | chasing the big)

25. The EPA task l Detailed diary of a single individual’s daily activity and location l Methods of collecting the data: »External Observer »Camera »Self-reporting –Paper diary –Handheld menu-driven diary –Spoken diary

26. Spoken Diary l From an utterance like “I am in the kitchen cooking spaghetti”, map that utterance into »Activity(cooking) »Location(kitchen) l Text abstraction l Technique »Build a grammar »Example

27. Sample Semantic Grammar ACTIVITY_LOCATION -> ACTIVITY' LOCATION' : CHAD(ACTIVITY',LOCATION'). ACTIVITY_LOCATION -> LOCATION' ACTIVITY' : CHAD(ACTIVITY',LOCATION'). ACTIVITY_LOCATION -> ACTIVITY' : CHAD(ACTIVITY', null). ACTIVITY_LOCATION -> LOCATION' : CHAD(null,LOCATION'). LOCATION -> IAM LOCx' : LOCx'. LOCATION -> LOCx' : LOCx'. IAM -> IAM1. IAM -> IAM1 just. IAM -> IAM1 going to. IAM -> IAM1 getting ready to. IAM -> IAM1 still. LOC2 -> HOUSE_LOC' : HOUSE_LOC'. LOC2 -> OUTSIDE_LOC' : OUTSIDE_LOC'. LOC2 -> WORK_LOC' : WORK_LOC'. LOC2 -> OTHER_LOC' : OTHER_LOC'. HOUSE_LOC -> kitchen : kitchen_code. HOUSE_LOC -> bedroom : bedroom_code. HOUSE_LOC -> living room : living_room_code. HOUSE_LOC -> house : house_code. HOUSE_LOC -> garage : garage_code. HOUSE_LOC -> home : house_code. HOUSE_LOC -> bathroom : bathroom_code. HOUSE_LOC -> den : den_code. HOUSE_LOC -> dining room : dining_room_code. HOUSE_LOC -> basement : basement_code. HOUSE_LOC -> attic : attic_code. OUTSIDE_LOC -> yard : yard_code.

28. Statistical Natural Language Parsing l Use unigram, bigram and trigram probabilities l Use Bayes’ rule to obtain these probabilities: P(A|B) = P(B|A) * P(A)/ P(B) l The formula P(“kitchen”|30121 Kitchen) is computed by determining the percentage of times the word “kitchen” appears in diary entries that have been transcribed in the category 30121 Kitchen. l P(30121 Kitchen) is the probability that a diary entry is of the semantic category 30121 Kitchen. l P(“kitchen”) is the probability that “kitchen” appears in any diary entry. l Bayes’ rule can be extended to take into account each word in the input string.

29. The Experiment l Digital Voice Recorder + Heart Rate Monitor »Heart rate monitor will beep if the rate changes by more than 15 beats per minute between measurements (every 2 minutes)

30. Subjects IDSexOccupationAgeEducation 1F Manages Internet Company 52Some College 2FGrocery Deli Worker18Some College 3MConstruction Worker35High School 4FDatabase Coordinator29Graduate Degree 5FCoordinator for Non-profit56Some College 6MUnemployed50High School 7MRetired76High School 8MDisabled62High School 9M Environment Technician 56Graduate Degree

31. Recordings Per Day

32. Heart Rate Change Indicator Tones and Subject Compliance SNumber of Tones Per Day (Avg.) % of Times Subject Made a Diary Entry Corresponding to a Tone 122.145% 241.829% 332.536% 433.055% 533.336% 615.640% 732.537% 826.022% 922.731%

33. Per Word Speech Recognition PPer Word Recognition Rate (%) 163 254 359 461 529 617 745 849 956

34. Semantic Grammar Location/Activity Encoding Precision and Recall Word Rec. Rate LocationActivity PrecisionRecallPrecisionRecall 16393708457 25491618155 35994699260 46186729562 52966157516 61755135114 74570507048 84971557954 95685708466 Av.48.17952.77948

35. Word Recognition Accuracy’s Effect on Semantic Grammar Precision and Recall

36. Statistical Processing Accuracy Activity Accuracy Location Accuracy Hand- transcribed 86.7%87.5% Using speech Recognition 48.3%49.0%

37. Word Recognition Affects Statistical Semantic Categorization Rec. Rate % LocationActivity Accuracy % 163 7769 254 4348 359 5663 461 71 529 2226 617 23 745 4238 849 4346 956 6852 Av.48.1 4849

38. Per Word Recognition Rate Versus Statistical Semantic Encoding Accuracy

39. Time, Activity, Location, Exertion Data Gathering Platform

40. Research Topics l Currently, guesses for the current activity and location are computed independently of each other »They are not independent! l Currently, guesses are based on the current utterance. »However, the current activity/location is not independent from previous activity/locations. l How do we fuse data from other sources (gps, beacons, heart rate monitor, etc.)?