1. Spoken Language Understanding for Conversational Dialog Systems
Michael McTear
University of Ulster
IEEE/ACL 2006 Workshop on Spoken Language Technology
Aruba, December 10-13, 2006

2. Overview Introductory definitions
Task-based and conversational dialog systems
Spoken language understanding
Issues for spoken language understanding
Coverage
Robustness
Overview of spoken language understanding
Hand-crafted approaches
Data-driven methods
Conclusions

3. Basic dialog system architecture
Speech
Recognition
Dialogue
Manager
Back
end
Language
Generation
Text to Speech
Synthesis
Audio
Spoken
Understanding
Words
Semantic
representation
Concepts
HMM
Acoustic
Model
N-Gram

4. Task-based Dialog Systems
Mainly interact with databases to get information or support transactions
SLU module creates a database query from user’s spoken input by extracting relevant concepts
System initiative: constrains user input
Keyword / keyphrase extraction
User-initiative: less constrained input
Call-routing: call classification with named entity extraction
Question answering

5. Conversational Dialog
AI (agent-based systems) e.g. TRIPS
User can take initiative, e.g. raise new topic, ask for clarification (TRIPS)
More complex interactions involving recognition of the user’s intentions, goals, beliefs or plans
Deep understanding of the user’s utterance, taking into account contextual information
Information State Theory, Planning Theory, User Modelling, Belief Modelling…
Simulated conversation e.g. CONVERSE
Conversational companions, chatbots, help desk
Does not require deep understanding
SLU involves identifying system utterance type and determining a suitable response

6. Defining Spoken Language Understanding
extracting the meaning from speech utterances
a transduction of the recognition result to an interpretable representation
Meaning (in human–computer interactive systems):
a representation that can be executed by an interpreter in order to change the state of the system
Bangelore et al. 2006

7. SLU for task based systems
a flight from Belfast to Malaga
uh I’d like uh um could you uh is there a flight from Bel- uh Belfast to um Gran- I mean Malaga
I would like to find a flight from Pittsburgh to Boston on Wednesday and I have to be in Boston by one so I would like a flight out of here no later than 11 a.m.
Topic: Flight
Origen: BFS
Destination: AGP

8. SLU for advanced conversational systems (TRIPS)
Interpretation requires intention recognition can we use a helicopter to get the people from: Abyss (request to modify plan) Barnacle (include sub-goal and suggest solution) Delta (extension of a solution)
Six possible interpretations with only change of city name
Requires reasoning about task and current context to identify most plausible interpretation
Requires more than concept spotting to identify structure and meaning of utterance as basis for reasoning

9. Role of syntax in deep understanding
List all employees of the companies who/which are based in the city centre
employees based in city centre
companies based in city centre
I would like to know where to mail the check.
I would like you to mail the check to me
I mail check
to ?where
You mail check
to me
Stop word removal will transform both strings to “mail check”

10. SLU for simulated dialog
C: Are you still a friend of XXX?
H: I am not sure if he has any real friends. He has achieved a lot and has left a lot of people behind.
C: You really don’t like the guy - well, at least that’s definite - I thought most people were just indifferent or numb by now. ?

11. Coverage Possible requirement:
“The system should be able to understand everything the user might say”
Predicting user input
Analysis of corpora and iterative design of hand-crafted grammars
Use of carefully designed prompts to constrain user input is constrained
Learning grammar from data

12. Robustness Characteristics of spontaneous spoken language
Disfluencies and filled pauses – not just errors, reflect cognitive aspects of speech production and interaction management
Output from speech recognition component
Words and word boundaries not known with certainty
Recognition errors
Approaches
Use of semantic grammars and robust parsing for concepts spotting
Data-driven approaches – learn mappings between input strings and output structures

13. Developing the SLU component
Hand-crafted approaches
Grammar development
Parsing
Data-driven approaches
Learning from data
Statistical models rather than grammars
Efficient decoding

14. Hand-crafting grammars
Parsing
Frame
Generation
Discourse
Processing DB
Query
parse tree
semantic frame
frame in context
SQL query
ASR
n-best list, word lattice, …
Traditional software engineering approach of design and iterative refinement
Decisions about type of grammar required
Chomsky hierarchy
Flat v hierarchical representations
Processing issues (parsing)
Dealing with ambiguity
Efficiency

15. Semantic Grammar and Robust Parsing: PHOENIX (CMU/CU)
The Phoenix parser maps input word strings on to a sequence of semantic frames.
named set of slots, where the slots represent related pieces of information.
each slot has an associated Context-Free Grammar that specifies word string patterns that match the slot
chart parsing with path pruning: e.g. path that accounts for fewer words is pruned
ASR
Semantic
Parser
word string
meaning
representation

16. Deriving Meaning directly from ASR output: VoiceXML
Uses finite state grammars as language models for recognition and semantic tags in the grammars for semantic parsing
ASR
meaning
representation
I would like a coca cola and three large pizzas with pepperoni and mushrooms {
drink: "coke",
pizza: {
number: "3",
size: "large",
topping: [
"pepperoni",
"mushrooms" ] }

17. Deep understanding Requirements for deep understanding
advanced grammatical formalisms
syntax-semantics issues
parsing technologies
Example: TRIPS
Uses feature-based augmented CFG with agenda-driven best-first chart parser
Combined strategy: combining shallow and deep parsing (Swift et al. )

18. Combined strategies: TINA (MIT)
Grammar rules include mix of syntactic and semantic categories
Context free grammar using probabilities trained from user utterances to estimate likelihood of a parse
Parse tree converted to a semantic frame that encapsulates the meaning
Robust parsing strategy
Sentences that fail to parse are parsed using fragments that are combined into a full semantic frame
When all things fail, word spotting is used

19. Problems with hand-crafted approaches
Hand-crafted grammars are
not robust to spoken language input
require linguistic and engineering expertise to develop if grammar is to have good coverage and optimised performance
time consuming to develop
error prone
subject to designer bias
difficult to maintain

20. Statistical modelling for SLU
SLU as pattern matching problem Given word sequence W, find semantic representation of meaning M that has maximum a posteriori probability P(M|W)
P(M): semantic prior model – assigns probability to underlying semantic structure
P(W|M): lexicalisation model – assigns probability to word sequence W given the semantic structure

21. Early Examples
CHRONUS (AT&T: Pieraccini et al, 1992; Levin & Pieraccini, 1995)
Finite state semantic tagger
‘Flat-concept’ model: simple to train but does not represent hierarchical structure
HUM (Hidden Understanding Model) (BBN: Miller et al, 1995)
Probabilistic CFG using tree structured meaning representations
Grammatical constraints represented in networks rather than rules
Ordering of constituents unconstrained - increases robustness
Transition probabilities constrain over-generation
Requires fully annotated treebank data for training

22. Using Hidden State Vectors (He & Young)
Extends ‘flat-concept’ HMM model
Represents hierarchical structure (right-branching) using hidden state vectors
Each state expanded to encode stack of a push down automaton
Avoids computational tractability issues associated with hierarchical HMMs
Can be trained using lightly annotated data
Comparison with FST model and with hand-crafted SLU systems using ATIS test sets and reference parse results

23. Which flights arrive in Burbank from Denver on Saturday?
Problem with long-distance dependency between ‘Saturday’ and ‘arrive’
‘Saturday’ associated with ‘FROMLOC’
Hierarchical model allows ‘Saturday’ to be associated with ‘ARRIVE’
Also: more expressive, allows sharing of sub-structures

24. SLU Evaluation: Performance
Statistical models competitive with approaches based on handcrafted rules
Hand-crafted grammars better for full understanding and for users familiar with system’s coverage, statistical model better for shallow and more robust understanding for naïve users
Statistical systems more robust to noise and more portable

25. SLU Evaluation: Software Development
“Cost of producing training data should be less than cost of hand-crafting a semantic grammar” (Young, 2002)
Issues
Availability of training data
Maintainability
Portability
Objective metrics? e.g. time, resources, lines of code, …
Subjective issues e.g. designer bias, designer control over system
Few concrete results, except …
HVS model (He & Young) can be robustly trained from only minimally annotated corpus data
Model is robust to noise and portable to other domains

26. Additional technologies
Named entity extraction
Rule-based methods: e.g. using grammars in form of regular expressions compiled into finite state acceptors (AT&T SLU system) – higher precision
Statistical methods e.g. HMIHY, learn mappings between strings and NEs – higher recall as more robust
Call routing
Question Answering

27. Additional Issues 1 ASR/SLU coupling Post-processing results from ASR
noisy channel model of ASR errors (Ringger & Allen)
Combining shallow and deep parsing
major gains in speed, slight gains in accuracy (Swift et al.)
Use of context, discourse history, prosodic information
re-ordering n-best hypotheses
determining dialog act based on combinations of features at various levels: ASR and parse probabilities, semantic and contextual features (Purver et al, Lemon)

28. Additional Issues 2
Methods for learning from sparse data or without annotation
e.g. AT&T system uses ‘active learning’ (Tur et al, 2005) to reduce effort of human data labelling – uses only those data items that improve classifier performance the most
Development tools e.g. SGStudio (Wang & Acero) – build semantic grammar with little linguistic knowledge

29. Additional Issues 3 Some issues addressed in poster session
Using SLU for:
Dialog act tagging
Prosody labelling
User satisfaction analysis
Topic segmentation and labelling
Emotion prediction

30. Conclusions 1 SLU approach is determined by type of application
finite state dialog with single word recognition
frame based dialog with topic classification and named entity extraction
advanced dialog requiring deep understanding
simulated conversation, …

31. Conclusions 2 SLU approach is determined by type of output required
syntactic / semantic parse trees
semantic frames
speech / dialog acts, …
intentions, beliefs, emotions, …

32. Conclusions 3 SLU approach is determined by
Deployment and usability issues
applications requiring accurate extraction of information
applications involving complex processing of content
applications involving shallow processing of content (e.g. conversational companions, interactive games)

33. Selected References
Bangalore, S., Hakkani-Tür, D., Tur, G. (eds), (2006) Special Issue on Spoken Language Understanding in Conversational Systems. Speech Communication 48.
Gupta, N., Tur, G., Hakkani-Tür, D., Bangalore, S., Riccardi, G., Gilbert, M. (2006) The AT&T Spoken Language Understanding System. IEEE Transactions on Speech and Audio Processing 14:1,
Allen, JF, Byron, DK, Dzikovska, O, Ferguson, G, Galescu, L, Stent, A. (2001) Towards conversational human-computer interaction. AI Magazine, 22(4):27–35.
Jurafsky, D. & Martin, J. (2000) Speech and Language Processing, Prentice-Hall
Huang, X, Acero, A, Hon, H-W. (2001) Spoken Language Processing: A Guide to Theory, Algorithm and System Development. Prentice-Hall