Enabling enactive interaction in virtualized experiences Stefano Tubaro and Augusto Sarti DEI – Politecnico di Milano, Italy

Nome relatore How do we enable virtualized enaction with incomplete sensory stimulation? Gaining enactive knowledge from a mediated experience Enactive Knowledge: information gained through perception-action interaction usually refers to an environment or a process (task) inherently multimodal as it requires the coordination of the various senses knowledge acquired in a mediated form is usually in symbolic or iconic form enactive knowledge is based on the active use of our senses direct and intuitive (based on experiece) often acquired without being aware of it Research on enaction is mostly focused on developing enactive multimodal interfaces

Nome relatore Realistic or plausible? Aside from videogames, where plausibility plays a dominant role, enactive knowledge acquisition is usually aimed at learning about reality. This is why mediated enaction usually relies on realism and sensorial transparency, which makes its applicability rather limited More promising (for massive use) is to accept virtualized enaction to be based on reduced sensory perception and develop methodologies that allow the user to develop reliable mappings between real and virtual enactive knowledge applications that are able to guess whether such mappings are correctly established in a closed-loop fashion

Nome relatore An example: the virtual dressing room The e-market of articles of clothing is slumping because it offers no virtual try-on at home The impact of applications such as virtual mirrors has been modest because of high cost; modest realism; and lack of multimodal interaction Research is needed for developing virtual try-on applications based on physically and perceptually plausible interactional rendering with reduced sensory perception (audio-visual), and limited realism, as long as we know how to render what is perceptually relevant for a virtual experience that can be instinctly remapped onto real life by the user correctly map haptic interaction onto audio-visual interaction (sensory substitution through cross-modal processing) correctly map real interaction onto virtual interaction exploiting the user’s experiential knowledge

Nome relatore An example: the virtual dressing room The mappings between experiential memory and virtual experience need be established in a closed-loop fashion by an intelligent HMI, which is able to read metacommunication cues carried by posture, gestures, gait, facial expression, voice prosody, etc. The HMI must become enactive and learn on the fly whether the virtual experience is being effectively remapped onto the real world

Nome relatore How do we enable enactive virtualized interpersonal interaction with incomplete sensory perception? Next step: enactive interpersonal interaction in virtual spaces Interpersonal interaction is based on a closed-loop exchange of symbolic or iconic information: Semantic information exchanged through speech, images, intentional gestures, display of clothes or specific objects,... meta-communication cues: body language (face expression, gestures, body posture, gait, etc.), voice prosody ad expression, tactile cues... Interactional meta-communication is purely enactive inherently multimodal instinctive (reduced awareness)

Nome relatore Issues in virtualized interactional experiences Two options for enactive interaction Interaction conveyed through signals metainformation to be captured, transferred and rendered as transparently and completely as possible e.g. video and audio 3D problems: lack of effective and inexpensive haptic sensors/display Interaction mediated by models e.g. 3D body models (avatars), interactional sounds, etc. needs: compensate for incomplete and imperfect sensing and displaying technology with intelligent sensors, affective and cross-modal processing, perceptual multimodal displays, etc. Again, the HMI must become enactive and be aware of the impact of the virtual experience onto the users

Nome relatore In perspective Natural human-machine interaction Multimodal interaction at multiple levels of abstracion: from physical to emotional Enactive machines Machines that learn from the interaction with humans Machines that learn from multimodal observations of human-to-human interaction Point of convergence between Research on intelligent sensing computer vision, audio and acoustic analysis and processing, pattern recognition, multimodal fusion,... Research on multimodal sensors and displays Research on behavioural psychology (social learning theory), and sociology