1. Dialogue Manager Senior Companion University of Sheffield March 2008

2. review

3. Senior Companion Dialogue Manager 2-tier planning where High-level plans built using the Cognitive Model Topic specific plans built using Dialogue Action Forms DAFS Dialogue Action Forms Augmented Transition networks Represent domain knowledge Modularizable to encapsulate sub-topics within a conversation domain Back channel response while working on more complete system utterances (hm, uh-huh)

4. Dialogue Manager Natural Language Understanding Dialogue history Ontology Photo app Generation NLU output of user utterance Semantic content of system utterance + emotion tags Cognitive Model Dialogue Manager Architecture

5. Dialogue Manager Dialogue Acts Named Entities Person Family Relations Sister Mother grandmother Location Celebration event Birthday Anniversary Wedding Christmas Pronoun references Semantic categories for User input objects (nouns) User input actions(verbs) Original user input string Modalities for user input NLP output Emotion Tags Planning System Model Cognitive Model Hierarchy of Semantic tags for Photos User instances Ontology Record of User and system utterances Photo/s id Photo tags Photo attributes Photo App Dialogue History DM input closeup

6. Dialogue Manager Number of output Utterances for each: Semantic Representation objects actions Dialogue Act Emotional content Output modality Expected user response category Cognitive Model Fission/Generation Speech Analysis Dialogue History Ontology Photo App DM output closeup

7. Dialogue Action Forms GUI editor for creating DAFs Composed of nodes and arcs containing tests and actions DAFs pre-stacked, but can be overidden by matching indexing terms (semantic classes, significant words) Essential for mixed-initiative conversation

8. SC Dialogue Manager Stack goodbye greeting Run Greeting DAF Pop greeting DAF Push photo DAF goodbye photo System start goodbye location event people date special memory goodbye location event date special memory Run photo DAF Run people DAF Pop people DAF People DAF

9. DAFs (1)

10. DAFs(2)

11. SC Output (1) Natural Language Generation Currently choosing from a selection of pre-defined surface forms. A Generation module to be developed (CU) To be shared with the Health and Fitness Companion To include input from the Cognitive model for emotion and conversational style (based on user profile) Will contain lists of alternative question realisations derived from corpora TTS Loquendo TTS Avatar AAA, CrazyTalk, Nabaztag (2-D)

12. more detail

13. Current Dialogue Manager Architecture Input Queue Input Manager NLUPhoto Watcher DAF Engine DAF Stack Output Queue Output ManagerNLG

14. Output of ASR In the format of Galaxy Communicator frame. For example: { c output :hypothesises ( {c hypothesis …} {c hypothesis …} … ) } {c hypothesis :ROName “Language Model Name” :Likelyhood -159.035736 :Confidence 0.758829 :String “nice one” :NumOfWords 2 :WordsInfo: ( {c word …} {c word …} ) } {c word :String “nice” :AcousticScore 3.791455 :Confidence 0.750377 :StartPoint 0 :EndPoint 42 :Language: “en-gb”}

15. NLU Input A textual string. For example: “Kate is the mother of Marry”. Output 1.GATE Annotation Set. A snippet output for the sentence: “Kate is the mother of Mary.” [AnnotationImpl: id=0; type=Token; features={category=NNP, kind=word, orth=upperInitial, length=4, string=Kate}; start=NodeImpl: id=0; offset=0; end=NodeImpl: id=1; offset=4, AnnotationImpl: id=1; type=SpaceToken; features={kind=space, length=1, string= }; start=NodeImpl: id=1; offset=4;end=NodeImpl: id=2; offset=5, AnnotationImpl: id=2; type=Token; features={category=VBZ, kind=word, orth=lowercase, length=2, string=is}; start=NodeImpl: id=2; offset=5; end=NodeImpl: id=3; offset=7, AnnotationImpl: id=3; type=SpaceToken; features={kind=space, length=1, string= }; start=NodeImpl: id=3; offset=7; end=NodeImpl: id=4; offset=8 ……]

16. NLU 2. A populated ontology in RDF format A snippet output for the sentence: “Kate is the mother of Mary.” ……………………… <hasName rdf:datatype="http://www.w3.org/2001/XMLSchema#string" >Kate <hasName rdf:datatype="http://www.w3.org/2001/XMLSchema#string" >someone2

17. DAFs DAFs are either triggered by Being on the top of the stack Through index term matching Each DAF has an associated set of terms associated with it (semantic categories)

18. Dialogue Manager Strategy Very simple for now, discusses variety of things in and about the photos People - their names, ages and relationship to the user Location of the photo and when it was taken and for what purpose DM strategy at present connects all photos that contain the same person and discusses the similarities and differences.

19. DAFs Created using a GUI editor Set of nodes and arcs A node represents a dialogue state Each arc contains a test and an associated action

20. DAF Editor Each node represents a dialogue state Each arc contains a test with an associated action

21. The main knowledge base will be an ontology which will hold all the world knowledge. There is another ontology which holds the current session knowledge. We chose ontologies to represent the knowledge because in the same file we have a structure, instances and logical rules. The tests invoke rules predefined in the ontology. If the rule evaluates to true, then the relative action is invoked, otherwise not. Like this, a person creating the rule needs not know anything on programming. The rules will look like... Person(?p) ^ hasAge(?p,?age) ^ greaterThan(?age,200) -> TooOld(?p) They can be written and tested in Protege using a graphical interface. DAF tests invoke protégé Rules

22. SWRL http://protege.cim3.net/cgi-bin/wiki.pl?SWRLLanguageFAQ SWRL is an acronym for Semantic Web Rule Language. It is intended to be the rule language of the Semantic Web. SWRL is based on the OWL Web Ontology Language. It allows users to write rules to reason about OWL individuals and to infer new knowledge about those individuals. SWRL rule contains an antecedent part, which is referred to as the body, and a consequent part, which is referred to as the head. Both the body and head consist of positive conjunctions of atoms SWRL rules are written in terms of OWL classes, properties and individuals SWRL supports Open World Reasoning One of the most powerful features of SWRL is its ability to support user-defined built-ins. A built-in is a predicate that takes one or more arguments and evaluates to true if the arguments satisfy the predicate. ･ Person(?p) ^ hasAge(?p, ?age) ^ swrlb:greaterThan(?age, 17) -> Adult(?p)

23. Not all user input can be structured If the NLU module cannot fit data into a structured form of the type we are expecting then the NLU module outputs an Annotation Set which has coordinates and properties for those coordinates.

24. Dialogue Manager uses the Knowledge Base to Access individual pieces of Knowledge Static system knowledge Dynamic user instances Make inferences about the knowledge * relies on the knowledge being structured

25. NLG Input The input of NLG is a logical representation which has three properties: domain, type and Value (optional). To generate a sentence of requesting some general information of a photo, following representation will be sent to NLG: Domain: PHOTO Type: REQUEST_GENERAL Value: EMPTY

26. NLG Output A predefined utterance from a template file. Below is a snippet of the template file. (The $ will be replaced by the value specified in the logical representation.) ……………… How old is $? What is $'s age? What is the age of $? $ is how old? How do you know $? What is your relationship with $? ……………..

27. additions to the DM (near future) Add emotions to the SC system Incorporate a cognitive model component Machine Learning for improving the dialogue strategy