Referring to Objects with Spoken and Haptic Modalities Frédéric LANDRAGIN Nadia BELLALEM & Laurent ROMARY LORIA Laboratory Nancy, FRANCE

Overview Context: Conception phase of the EU IST/MIAMM project (Multidimensional Information Access using Multiple Modalities - with DFKI, TNO, Sony, Canon) –Study of the design factors for a future haptic PDA like device Background: Interpretation of natural language and spontaneous gestures. –A model of contextual interpretation of multimodal referring expressions in visual contexts. Objective: To show the possible extension of the model to an interaction mode including tactile and kinesthetic feedback (henceforth “Haptic”)

MIAMM - wheel mode

MIAMM - axis mode

MIAMM - galaxy mode

The use of perceptual grouping Reference domains and visual contexts « these three objects »  {,, }

The use of perceptual grouping Reference domains and visual contexts « these three objects »  {,, } « the triangle »  { } « the two circles »  {, } The use of salience

Multimodal fusion architecture reference domain(s) history task module visual module dialogue manager request domains request domains request domains referential expression referent history language module under- specification user model referential gesture

Haptics and deixis Haptic gestures can take three classical functions of gesture in man-machine interaction: –semiotic function: ‘select this object’ –ergotic function: ‘reduce the size of this object’ –epistemic function: ‘save the compliance of this object’ How can the system identify the function(s)? –linguistic clues (referential expression, predicate) –task indications (possibilities linked to a type of objects, “affordances”) Deictic role: to make the object salient, whatever the function, in order to focus the addressee’s attention on it.

Haptics and perceptual grouping Interest: formalism for the focalization on a subset of objects Grouping factors: –objects which have similar haptic (in particular, tactile) properties (shape, consistency, texture) –objects that have been browsed by the user (the elements of such a group are ordered) –objects that are stuck together, parts of a same object...

Haptics and perceptual domains Can visual and tactile perceptions work together? –simultaneous visual and tactile perception implies the same world of objects (and synchronized feedbacks) –a referring expression can be interpreted in visual context or in tactile context How can the system identify the nature of perception? –for immediate references, the visual context gives the reference domain and haptics gives the starting point in it –for ieterating references, each type of context can provide the reference domain (so both hypotheses must be tested)

Haptics and dialogue history Interpretations that require an ordering within the reference domain: ‘the first one’, ‘the next one’, ‘the last one’ –in visual perception, guiding lines can be helpful (if none, an order can always be built with the reading direction) –in haptic perception, the only criterion can be the manipulation order Some referring expressions that do not need an order may be interpreted in the haptic manipulation history –‘the big one’ (in the domain of browsed objects) –‘them’ (the most pressured objects)

Architecture for speech-haptic referring reference domain(s) history task module visuo- tactile module dialogue manager request domains request domains request domains referent history language module under- specification user model haptic gesture referential expression

Summary What does not change from deictic to haptic: –the status of speech and gesture in the architecture –the repartition of information among speech and gesture –the need of reference domains –the use of salience and the use of orderings in domains –the algorithms for the exploitation of all these notions What does change: –some unchanged notions can have more than one cause –objects must be browsed to be grouped in a haptic domain –one aspect of the architecture: the visual perception module becomes the visuo-tactile perception module

Application - MMIL design MMIL - Multimodal Interface Language –Unique communication language within the MIAMM architecture –XML based representation embedded in a web service protocol (SOAP) –Oriented towards multimodal content representation (cf. Joint ACL/SIGSEM and TC37/SC4 initiative) Example: VisHapTac notification to Dialogue Manager

Overall structure e0e0 set 1 s1s1 s2s2 set 2 s 25 … description s 2-1 s 2-2 s 2-3 inFocus inSelection Visual haptic state Participant setting Sub-divisions

MMIL format for VisHapTac - 1 HGState galaxy <tempSpan startPoint=“ T14:12:06” endPoint=“ T14:12:13”/> …

MMIL format for VisHapTac - 2 … Let it be set inFocus Lady Madonna … inSelection Revolution 9 …

Future work Within the dialogue manager module, domains may be confronted, using a relevance criterion The way the linguistic contraints of the referring expression apply in the different domains may be such a criterion. Validation in the MIAMM architecture The transition from deictic to haptic may not be an additional cost for the development of a dialogue system, both from the architecture point of view and the dialogue management point of view.