Evolvable dialogue systems Milica Gašić Dialogue Systems Group

What constitutes a spoken dialogue system? Understanding the user Deciding what to say back Conducting a conversation beyond simple voice commands or question answering

What breaks a spoken dialogue system? Speech recognition errors Not keeping track of what happened previously Need to hand-craft a large number of rules Poor decisions User's request is not supported ...

Machine learning Allows learning from data Automates the development process Uses sophisticated statistical models

Machine learning for dialogue modelling

Data Abundant data / unlabeled data / out-of-domain data Limited data / labeled data / in-domain data Noisy data

Machine learning methods Supervised learning Unsupervised learning Reinforcement learning

Architecture Tree-base architecture (VoiceXML) Modular architecture End-2-end neural network architecture

Prediction What constitutes a good dialogue? How to evaluate dialogue systems? Which measures to use?

Why Machine Learning for dialogue modelling? Cheaper development / avoids handcrafting Robust to errors / models that propagate uncertainty Improves with the time of use / support adaptation According to Darwin's Origin of Species: “It is not the most intellectual of the species that survives; it is not the strongest that survives; but the species that survives is the one that is able best to adapt and adjust to the changing environment in which it finds itself.”

Language Understanding Dialogue management Can I have Korean food? Speech Recognition Language Understanding inform(food=Korean) Belief tracker Policy Optimisation Dialogue Management Knowledge Base A system that allows human computer interaction using speech as the primary medium. Instant and human-like acquisition of information. Speech Synthesis Language Generation inform(name=“Little Seoul”, food=Korean) Little Seoul is a nice Korean restaurant

What is adaptation? Testing in different conditions to training Using a small amount of in-domain data to retrain a system In dialogue modelling Different noise conditions Different user Different domain

Problem: Dialogue systems adaptation – new domain Data: Small amount of in-domain data Model: Method: reinforcement learning, Gaussian processes, Bayesian committee machine, multi-agent learning Architecture: distributed, committee model Predictions: Faster convergence

Machine Learning Theory: Reinforcement learning observations ot Dialogue system reward rt belief states bt action at

Machine Learning Theory: Reinforcement learning observations ot Dialogue system reward rt belief states bt action at Policy

Reinforcement learning in practice

Machine Learning Theory: Gaussian process

Machine Learning Theory: Gaussian process

Machine Learning Theory: Gaussian process

Machine Learning Theory: Gaussian process

Machine Learning Theory: Gaussian process

Q-function as a Gaussian process

Gaussian process Reinforcement learning observations ot Dialogue system reward rt Estimate Q-function action at

Other suitable models Desirable properties Uncertainty estimate Incorporation of the prior knowledge Other possible estimators include Bayesian Neural Networks Need sufficient training data The unsertanty estimates can be obtained using dropout, vatiational inference aproach, recent developments for batch gradient based MCMC. Dropout – remove some hidden untis randomly Variational approach – use another probility distibution to approximate the posterior over parameters Batch versions of MCMC – We use MCMC to approximate the posterior but not the whole training set but in batch

Problem 1: Build dialogue systems from limited data Limited training data available for new domain(s) Solution: Use distributed architecture to train a model from disperse data Essential ingredients: Gaussian process prior

Distributed dialogue management DH +DR MV MV MV MR MH DR DH restaurant restaurant restaurant restaurant restaurant DH DR DL restaurant DH DR DL MR DH DR DL MR DH DR DL MR DH DR DL MR DH DR DL MR MR hotel MH hotel ML laptops MH hotel ML laptops MH hotel ML laptops MH hotel ML laptops MH hotel ML laptops MH ML laptops Committee Model Committee Model Committee Model Committee Model Committee Model Committee Model

Results Gašić et al, Distributed dialogue policies for multi-domain statistical dialogue management, ICASSP, 2015

Problem 2: Domain knowledge is not shared Estimates of Q-function in different domains do not share knowledge Solution: Reuse knowledge obtained in one domain for another domain Essential ingredients: Bayesian committee machine

Committee of dialogue managers DR MR DH MH ML DL Committee Model restaurant restaurant restaurant restaurant restaurant DH DR DL restaurant DH DR DL MR DH DR DL MR DH DR DL MR DH DR DL MR DH DR DL MR MR hotel MH hotel ML laptops MH hotel ML laptops MH hotel ML laptops MH hotel ML laptops MH hotel ML laptops MH ML laptops Committee Model Committee Model Committee Model Committee Model Committee Model Committee Model

Results

Results Gašić et al, Policy Committee for adaptation in multi-domain spoken dialogue systems, ASRU, 2015

Problem 3: Learning from out-of-domain data Estimates of Q-function do not learn from out-of-domain data Solution: Perform learning using both in-domain and out-of-domain data Essential ingredients: Multi-agent learning

Multi-agent learning DR MR DH MH ML DL Committee Model

Multi-agent results Gašić et al, Multi-agent learning in multi-domain spoken dialogue systems, NIPS SLUI workshop, 2015 , 2015

Summary: Adaptation Learning from limited data for a new domain Model for new domain uses knowledge from models for other domains Model for new domain learns from out-of-domain dialogues

Evolution roadmap I feel sad… I’ve got a cold what do I do? Dialogue depth (complexity) Tell me a joke. Single domain systems Extended systems Multi-domain systems Open domain systems What is influenza? Dialogue breadth (coverage)

Properties of natural conversation Less task focused, but more open-ended Beyond speech recognition – incorporates sentiment, gesture and emotion Long-term conversation Incremental operation …

Conclusions Machine learning enables building evolving dialogue systems Gaussian processes in combination with reinforcement learning allow domain adaptation From here we can Model more natural conversation And address new challenging tasks

Dialogue Systems Group Steve Young Lina Rojas Barahona Stefan Ultes Tsung-Hsien Wen Pei-Hao Su Nikola Mrkšić Pawel Budzianowski