1. Spoken Dialog System

9. The Fifth Dialog State Tracking Challenge
DSTC5
The Fifth Dialog State Tracking Challenge

10. CHALLENGE OVERVIEW
For each turn in a given sub-dialog ,the tracker should fill out a frame of slot-value pairs considering all dialog history prior to the turn
· Main Task
The performance of a tracker will be evaluated by comparing its outputs with reference annotations
In the development phase, participants will be provided with a training set of English dialogs and a development set of Chinese dialogs with manual annotations over frame structures
· Pilot Task

12. CHALLENGE OVERVIEW Spoken language understanding (SLU):
Tagging a given utterance with speech acts and semantic slots.
Speech act prediction (SAP):
Predicting the speech act of the next turn imitating the policy of one speaker
Spoken language generation (SLG):
Generating a response utterance for one speaker by using the corresponding speech act and semantic slot information.
End-to-end system (EES):
Developing an end-to-end system by pipe- lining and/or combining different SLU,SAP and SLG systems.
· Main Task
· Pilot Task

13. CHALLENGE OVERVIEW · Data
For the purposes of the DSTC5 Challenge, the TourSG corpus has been divided in the following three parts:
Training set:
manual transcriptions and annotations at both utterance and sub-dialog levels will be provided for 35 English dialogs
Development set:
2 Chinese dialogs with 5-best MT results in English also with word alignment information
Test set:
for the main task, manual transcriptions and their aligned MT results will be provided for 10 Chinese dialogs for evaluating the trackers. In the case of the pilot tasks, 8 Chinese dialogs will be used for evaluating each task

15. CHALLENGE OVERVIEW ·Evaluation Main Task ： Accuracy ：
Fraction of segments in which the tracker’s output is equivalent to the gold standard frame structure
Precision：
Fraction of slot-value pairs in the tracker’s outputs that are correctly filled
Recall：
Fraction of slot-value pairs in the gold standard labels that are correctly filled
F-measure ：
The harmonic mean of precision andrecall

16. CHALLENGE OVERVIEW ·Evaluation Pilot Task ：
Spoken Language Understanding (SLU):
Speech Act Prediction (SAP):
Spoken Language Generation (SLG):

17. CHALLENGE OVERVIEW ·Evaluation Pilot Task ：
Spoken Language Understanding (SLU):
Speech Act Prediction (SAP):
Spoken Language Generation (SLG):

20. DIALOG STATE TRACKING WITH ATTENTION-BASED
DIALOG STATE TRACKING WITH ATTENTION-BASED SEQUENCE-TO-SEQUENCE LEARNING

21. we select for each tracker unreliable slot types where it poorly predicts the slot values, using the training and development data.
The rule-based trackers have better performances in slot types with concrete values such as “PLACE” and “STATION” while the attention-based tracker has a better performance in slot types with abstract values such as “INFO” and “TYPE OF PLACE.”