1. Multimodal Communication in the Staging Virtual farm Patrizia Paggio and Bart Jongejan Center for Sprogteknologi MUMIN workshop Helsinki 2002

2. Paggio and Jongejan - Helsinki ‘022 The Staging project (www.staging.dk) Interdisciplinary Danish project: nature and use of 3D applications populated with autonomous agents. CST’s work: multimodal communication components of a 3D virtual farm. Focus: multimodal integration, mixed-initiative dialogue, interaction between dialogue and other behaviours.

3. Paggio and Jongejan - Helsinki ‘023 The Staging VE The VE l is in charge of simulating the world l provides the agents with sensory information l processes requests from the agents (move objects, produce sounds, play animations) Staging VE developed at CVMT (Aalborg University) CST has developed a mock-up for testing purposes.

4. Paggio and Jongejan - Helsinki ‘024 Agents Agents carry out behaviours l in reaction to external stimuli and according to their inner state (hunger, tiredness…) l based on strength of activation level Engaging in a dialogue with the user’s avatar is also a behaviour. Dialogue behaviour has strong degree of activation for the farmer agent.

5. Paggio and Jongejan - Helsinki ‘025 The Aalborg VE

6. Paggio and Jongejan - Helsinki ‘026 The CST farm Her skal vises et billede af vores VE

8. Paggio and Jongejan - Helsinki ‘028 Multimodal communication User can interact with agents via various devices: microphone, keyboard, touch screen, data glove. Commercial speech technology, dedicated gesture recogniser (Karin Husballe Munk at CVMT). Speech can be combined with deictic, iconic and turn- taking gestures (Cassell and Prevost 1996). Gestures and speech merged by multimodal parser.

9. Multimodal integration Speech recognition Parsing Semantic mapping Communication management Chart initialisation SpeechHand movements Gesture recognition Action pointing, size turn- taking

10. Paggio and Jongejan - Helsinki ‘0210 More integration Gesture and word are paired: Feed that cow$1|cow Gesture adds information to lexicon entry. l Word and gesture must be (nearly) synchronous l Syntactic constraints:  deictic (pointing) requires noun or pronoun  iconic (size) requires noun l Semantic constraints:  semantic types must be compatible

11. Paggio and Jongejan - Helsinki ‘0211 Example Feed that cow$1|animal. pointgesture := $ {act=request, predicate=feed, arg3={reln=animal, semtype=animal, objectid=cow$1}} reln and object type unified, semtype compatible, objectid added.

12. Paggio and Jongejan - Helsinki ‘0212 Contradiction example Feed that cow$1|apple. {act=request, predicate=feed, arg3={reln=animal, semtype=animal, objectid=cow$1}} gesture and noun semantic types incompatible; only the interpretation provided by the gesture is compatible with the semantics of the predicate and survives.

13. Paggio and Jongejan - Helsinki ‘0213 Examples Deictic gestures U: Feed an animal, please. A: Which animal shall I feed? U: Take that cow (+ pointing) Iconic gestures U: Feed the sheep, please. A: Which food shall I take? U: The small apple (+ size) Turn-giving and taking gestures U: Hi (+give turn) A: Shall we...

14. Paggio and Jongejan - Helsinki ‘0214 The Communication Manager l Interprets user’s dialogue moves l Builds dialogue trees l Interprets references not resolved by gestures l Decides agent’s dialogue moves based on preceding dialogue and on changes in the VE Dialogue goals arising from scenario combined with dialogue obligations created by preceding dialogue.

15. Paggio and Jongejan - Helsinki ‘0215 Dialogue goals Dialogue goals are created based on domain-specific action templates (Badler et al 1999). A template specifies actions with related semantic arguments, corresponding attribute name in the semantic representation, necessary preconditions. FeedAction(Topic=Feed, Animal=, Food=, Tool=, Precondition=Hungry(Animal))

16. Paggio and Jongejan - Helsinki ‘0216 Example: feed action U: Hi come here A: Okay, I’ll do it U: Feed an animal. A: Which animal shall I take? U: That cow$1|cow. A: Which food shall I take? U: (Take) a small$|small apple. A: Which tool shall I take? U: Take the pitchfork. A: Okay, I’ll do it.

17. Paggio and Jongejan - Helsinki ‘0217 Example: precondition not met U: Give that brown cow$2|cow an apple, please.... A: The cow is not hungry.

18. Paggio and Jongejan - Helsinki ‘0218 Example: agent initiative A: Shall I feed the brown cows and the sheep? U: Yes, give the animals a carrot. A: Which tool shall I take? U: The pitchfork. A: Okay, I’ll do it.

19. Paggio and Jongejan - Helsinki ‘0219 Dialogue obligations Set of condition/obligation pairs model valid speech act sequences. E.g.: Request/Accept, Reject Whque/Answer, Inform Used to l produce a correct reaction to a user move l interpret a user move as either closing a dialogue segment or opening a new one

20. Paggio and Jongejan - Helsinki ‘0220 Dialogue trees requestU: Give the white cow an apple please. whqueA: Which tool shall I use? whqueU: Where is the pitchfork ? informA: The pitchfork is in front of the tree. requestU: Take the pitchfork then. acceptA: Okay, I’ll do it.

21. Paggio and Jongejan - Helsinki ‘0221 Relaxing the rules Condition/obligation pairs do not always fit. Speech acts can be implied: A: Hi U: Feed the animals please They can be coerced: U: Feed an animal. A: Which animal shall I take? U: Feed the brown cow then.

22. Paggio and Jongejan - Helsinki ‘0222 Conclusions Staging has made an initial attempt at giving an agent multimodal dialogue abilities to allow for mixed- initiative dialogues. Future research: l more advanced gesture recognition l better understanding of how gestures and speech can complement each other l repairs and self-repairs l interaction between dialogue and other behaviours

23. Paggio and Jongejan - Helsinki ‘0223 FILM