1. A Cross-modal Electronic Travel Aid Device F. Fontana, A. Fusiello, M. Gobbi, V. Murino, D. Rocchesso, L. Sartor, A. Panuccio. Università di Verona Dipartimento di Informatica

2. Andrea Fusiello Mobile HCI 20022 Overview n Motivations n The system n Visual analisys n Auditory display n Work in progress

3. Andrea Fusiello Mobile HCI 20023 Motivations n Humans use several senses simultaneously to explore and experience the environment. n Multimodal displays can enhance the user experience and the sense of engagement. n The redundancy of our sensory system can be exploited in order to choose the display that is the most convenient for a given application. n Our interest: exploring new ways of transferring information across different sensorial modalities, especially in the context of interactive systems.

4. Andrea Fusiello Mobile HCI 20024 n Cross-modality: methods to perform analysis tasks using one modality, and synthesis (display) tasks using another modality. n Examples: give auditory display to visual information, visualize auditory scenes, and sonify haptic sensations. n Design problem: how to render a given percept using a certain modality, the latter possibly being not the most obvious for the stimulus at hand.

5. Andrea Fusiello Mobile HCI 20025 A cross-modality instance n Auditory display of visual information –More effective HC interfaces –Aid devices for visually impaired people n Auditory display needs sonification of the (visual) objects being displayed. n Sonification is the acoustic analogous of visualization.

6. Andrea Fusiello Mobile HCI 20026 n n The blind person is equipped with a stereo camera and earphones. n n He/she uses a laser pointer as a cane. n n The system computes the 3D position of the laser spot and sonifies this piece of information The system

7. Andrea Fusiello Mobile HCI 20027 Visual analisys n Track the position of the laser spot in two images n Compute the 3D position of the laser spot by triangulation

8. Andrea Fusiello Mobile HCI 20028 Stereopsis: the concept The same 3D point projects onto two different pixels: the difference is the disparity, which is related to the depth of the point.

9. Andrea Fusiello Mobile HCI 20029 Laser spot tracking n Red band-pass filter n Brightness normalization n Temporal averaging n Thresholding n Size filter n Epipolar constraint n Kalman filter

10. Andrea Fusiello Mobile HCI 200210 Auditory display n Sonification: the most important visual objects are associated to sounds that “label” such objects. n Visual objects are: –The laser spot –Disparity blobs n For disparity blobs we sonify their area, using pitch and loudness.

11. Andrea Fusiello Mobile HCI 200211 n Spatialization: sounds are positioned in a virtual scenario, and displayed in 3D –Relative angular position –Relative distance

12. Andrea Fusiello Mobile HCI 200212 Spatial attributes n Distance is conveyed by processing sounds as if they were located inside a tube n n Angular position: – –Interaural delay – –Head diffraction (low pass)

13. Andrea Fusiello Mobile HCI 200213 The prototype

14. Andrea Fusiello Mobile HCI 200214 Work in progress n Empirical evaluation n Porting to PocketPC n Sketch based interface n Other sonification modes (global) n Looking for other interesting applications of cross-modality

15. Andrea Fusiello Mobile HCI 200215 Questions?

16. Andrea Fusiello Mobile HCI 200216 Brown and Duda model n Interaural distance  time delay n Head diffraction  low pass filter on the opposite hear n Brown&Duda, IEEE Tr. Speech and Audio, 6(5) 1998.