1. Evolvable dialogue systems
Milica Gašić
Dialogue Systems Group

2. What constitutes a spoken dialogue system?
Understanding the user
Deciding what to say back
Conducting a conversation beyond simple voice commands or question answering

3. What breaks a spoken dialogue system?
Speech recognition errors
Not keeping track of what happened previously
Need to hand-craft a large number of rules
Poor decisions
User's request is not supported
...

4. Machine learning Allows learning from data
Automates the development process
Uses sophisticated statistical models

5. Machine learning for dialogue modelling

6. Data Abundant data / unlabeled data / out-of-domain data
Limited data / labeled data / in-domain data
Noisy data

7. Machine learning methods
Supervised learning
Unsupervised learning
Reinforcement learning

8. Architecture Tree-base architecture (VoiceXML) Modular architecture
End-2-end neural network architecture

9. Prediction What constitutes a good dialogue?
How to evaluate dialogue systems?
Which measures to use?

10. Why Machine Learning for dialogue modelling?
Cheaper development / avoids handcrafting
Robust to errors / models that propagate uncertainty
Improves with the time of use / support adaptation
According to Darwin's Origin of Species: “It is not the most intellectual of the species that survives; it is not the strongest that survives; but the species that survives is the one that is able best to adapt and adjust to the changing environment in which it finds itself.”

11. Language Understanding
Dialogue management
Can I have Korean food?
Speech Recognition
Language Understanding
inform(food=Korean)
Belief tracker
Policy Optimisation
Dialogue Management
Knowledge Base
A system that allows human computer interaction using speech as the primary medium.
Instant and human-like acquisition of information.
Speech Synthesis
Language Generation
inform(name=“Little Seoul”, food=Korean)
Little Seoul is a nice Korean restaurant

12. What is adaptation? Testing in different conditions to training
Using a small amount of in-domain data to retrain a system
In dialogue modelling
Different noise conditions
Different user
Different domain

13. Problem: Dialogue systems adaptation – new domain
Data:
Small amount of in-domain data
Model:
Method: reinforcement learning, Gaussian processes, Bayesian committee machine, multi-agent learning
Architecture: distributed, committee model
Predictions:
Faster convergence

14. Machine Learning Theory: Reinforcement learning
observations ot
Dialogue system
reward rt
belief states bt
action at

15. Machine Learning Theory: Reinforcement learning
observations ot
Dialogue system
reward rt
belief states bt
action at
Policy

16. Reinforcement learning in practice

17. Machine Learning Theory: Gaussian process

18. Machine Learning Theory: Gaussian process

19. Machine Learning Theory: Gaussian process

20. Machine Learning Theory: Gaussian process

21. Machine Learning Theory: Gaussian process

22. Q-function as a Gaussian process

23. Gaussian process Reinforcement learning
observations ot
Dialogue system
reward rt
Estimate
Q-function
action at

24. Other suitable models Desirable properties Uncertainty estimate
Incorporation of the prior knowledge
Other possible estimators include
Bayesian Neural Networks
Need sufficient training data
The unsertanty estimates can be obtained using dropout, vatiational inference aproach, recent developments for batch gradient based MCMC.
Dropout – remove some hidden untis randomly
Variational approach – use another probility distibution to approximate the posterior over parameters
Batch versions of MCMC – We use MCMC to approximate the posterior but not the whole training set but in batch

25. Problem 1: Build dialogue systems from limited data
Limited training data available for new domain(s)
Solution:
Use distributed architecture to train a model from disperse data
Essential ingredients:
Gaussian process prior

26. Distributed dialogue management
DH +DR MV MV MV MR MH DR DH
restaurant
restaurant
restaurant
restaurant
restaurant DH DR DL
restaurant DH DR DL MR DH DR DL MR DH DR DL MR DH DR DL MR DH DR DL MR MR
hotel MH
hotel ML
laptops MH
hotel ML
laptops MH
hotel ML
laptops MH
hotel ML
laptops MH
hotel ML
laptops MH ML
laptops
Committee
Model
Committee
Model
Committee
Model
Committee
Model
Committee
Model
Committee
Model

27. Results
Gašić et al, Distributed dialogue policies for multi-domain statistical dialogue management, ICASSP, 2015

28. Problem 2: Domain knowledge is not shared
Estimates of Q-function in different domains do not share knowledge
Solution:
Reuse knowledge obtained in one domain for another domain
Essential ingredients:
Bayesian committee machine

29. Committee of dialogue managersDR MR DH MH ML DL
Committee
Model
restaurant
restaurant
restaurant
restaurant
restaurant DH DR DL
restaurant DH DR DL MR DH DR DL MR DH DR DL MR DH DR DL MR DH DR DL MR MR
hotel MH
hotel ML
laptops MH
hotel ML
laptops MH
hotel ML
laptops MH
hotel ML
laptops MH
hotel ML
laptops MH ML
laptops
Committee
Model
Committee
Model
Committee
Model
Committee
Model
Committee
Model
Committee
Model

30. Results

31. Results
Gašić et al, Policy Committee for adaptation in multi-domain spoken dialogue systems, ASRU, 2015

32. Problem 3: Learning from out-of-domain data
Estimates of Q-function do not learn from out-of-domain data
Solution:
Perform learning using both in-domain and out-of-domain data
Essential ingredients:
Multi-agent learning

33. Multi-agent learning DR MR DH MH ML DL
Committee
Model

34. Multi-agent results
Gašić et al, Multi-agent learning in multi-domain spoken dialogue systems, NIPS SLUI workshop, 2015
, 2015

35. Summary: Adaptation Learning from limited data for a new domain
Model for new domain uses knowledge from models for other domains
Model for new domain learns from out-of-domain dialogues

36. Evolution roadmap I feel sad… I’ve got a cold what do I do?
Dialogue depth (complexity)
Tell me a joke.
Single domain systems
Extended systems
Multi-domain systems
Open domain systems
What is influenza?
Dialogue breadth (coverage)

37. Properties of natural conversation
Less task focused, but more open-ended
Beyond speech recognition – incorporates sentiment, gesture and emotion
Long-term conversation
Incremental operation …

38. Conclusions
Machine learning enables building evolving dialogue systems
Gaussian processes in combination with reinforcement learning allow domain adaptation
From here we can
Model more natural conversation
And address new challenging tasks

39. Dialogue Systems Group
Steve Young
Lina Rojas Barahona
Stefan Ultes
Tsung-Hsien Wen
Pei-Hao Su
Nikola Mrkšić
Pawel Budzianowski