A Cross-modal Electronic Travel Aid Device F. Fontana, A. Fusiello, M. Gobbi, V. Murino, D. Rocchesso, L. Sartor, A. Panuccio. Università di Verona Dipartimento di Informatica

Andrea Fusiello Mobile HCI Overview n Motivations n The system n Visual analisys n Auditory display n Work in progress

Andrea Fusiello Mobile HCI Motivations n Humans use several senses simultaneously to explore and experience the environment. n Multimodal displays can enhance the user experience and the sense of engagement. n The redundancy of our sensory system can be exploited in order to choose the display that is the most convenient for a given application. n Our interest: exploring new ways of transferring information across different sensorial modalities, especially in the context of interactive systems.

Andrea Fusiello Mobile HCI n Cross-modality: methods to perform analysis tasks using one modality, and synthesis (display) tasks using another modality. n Examples: give auditory display to visual information, visualize auditory scenes, and sonify haptic sensations. n Design problem: how to render a given percept using a certain modality, the latter possibly being not the most obvious for the stimulus at hand.

Andrea Fusiello Mobile HCI A cross-modality instance n Auditory display of visual information –More effective HC interfaces –Aid devices for visually impaired people n Auditory display needs sonification of the (visual) objects being displayed. n Sonification is the acoustic analogous of visualization.

Andrea Fusiello Mobile HCI n n The blind person is equipped with a stereo camera and earphones. n n He/she uses a laser pointer as a cane. n n The system computes the 3D position of the laser spot and sonifies this piece of information The system

Andrea Fusiello Mobile HCI Visual analisys n Track the position of the laser spot in two images n Compute the 3D position of the laser spot by triangulation

Andrea Fusiello Mobile HCI Stereopsis: the concept The same 3D point projects onto two different pixels: the difference is the disparity, which is related to the depth of the point.

Andrea Fusiello Mobile HCI Laser spot tracking n Red band-pass filter n Brightness normalization n Temporal averaging n Thresholding n Size filter n Epipolar constraint n Kalman filter

Andrea Fusiello Mobile HCI Auditory display n Sonification: the most important visual objects are associated to sounds that “label” such objects. n Visual objects are: –The laser spot –Disparity blobs n For disparity blobs we sonify their area, using pitch and loudness.

Andrea Fusiello Mobile HCI n Spatialization: sounds are positioned in a virtual scenario, and displayed in 3D –Relative angular position –Relative distance

Andrea Fusiello Mobile HCI Spatial attributes n Distance is conveyed by processing sounds as if they were located inside a tube n n Angular position: – –Interaural delay – –Head diffraction (low pass)

Andrea Fusiello Mobile HCI The prototype

Andrea Fusiello Mobile HCI Work in progress n Empirical evaluation n Porting to PocketPC n Sketch based interface n Other sonification modes (global) n Looking for other interesting applications of cross-modality

Andrea Fusiello Mobile HCI Questions?

Andrea Fusiello Mobile HCI Brown and Duda model n Interaural distance  time delay n Head diffraction  low pass filter on the opposite hear n Brown&Duda, IEEE Tr. Speech and Audio, 6(5) 1998.