1. The 11 th International one-month Summer Workshop on Multimodal Interfaces Aug 10 th – Sept 4 th 2015, Mons, Belgium A. Camurri, M. Mancini, G. Volpe, F. Odone Analysis of the qualities of human movement in individual performances project2@enterface.net

2. Université de Mons Project objectives  Investigation :  human movements  movements expressive qualities  sensory substitution techniques  Application :  enable blind people to perceive bodily movement Context 2 Project 2 |Analysis of the qualities of human movement in individual performances

3. Université de Mons Movement Features Considered 3 Project 2 | Analysis of the qualities of human movement in individual performances We are considering three different kinds of characteristics of the dynamic of a movement  Is it fluid or rigid?  With how much energy is the subject performing the movement?  Could we spot an impulsive behaviour within the execution of the movement?

4. Université de Mons Extraction Techniques (accelerometers) 4 Project 2 | Analysis of the qualities of human movement in individual performances Starting from the data available from the accelerometers we extract some characteristics of the dynamics of the movements analysed

5. Université de Mons Extraction Techniques (video sequences) 5 Project 2 | Analysis of the qualities of human movement in individual performances Evaluating the behaviour of the optical flow between each pair of frames of the sequence considered, by analysing the evolution of the resulting histogram of the optical flow

6. Université de Mons Interactive Sonification 6 Project 2 | Analysis of the qualities of human movement in individual performances Definition: data exploration by interactively manipulating the data’s transformation into sound Data from accelerometers, webcam Transformation

7. Université de Mons System architecture 7 Project 2 | Analysis of the qualities of human movement in individual performances

8. Université de Mons Experiment: Testing and Validation of the System 8 Project 2 | Analysis of the qualities of human movement in individual performances Investigation of the user’s capabilities and sensibility in recognizing individual movement qualities by means of the auditory modality

9. Université de Mons Experiment: Testing and Validation of the System 9 Project 2 | Analysis of the qualities of human movement in individual performances Q1: Is it possible for the user to understand the mapping between his movement qualities (fluid – rigid, low energy – high energy, impulsive movements) and the associated sonifications while exploring the feature space? Q2: Given a particular movement feature, which is the sound that represents it the most? Q3 Is it possible for the user to distinguish the proposed movement qualities only through the auditory domain ?

10. Université de Mons Experiment (I) 10 Project # | Title of the project

11. Université de Mons Equipment 11 Project # | Title of the project

12. Université de Mons Experiment (II) 12 Project # | Title of the project

13. Université de Mons Collected Feedbacks 13 Project # | Title of the project not at all very much

14. Université de Mons Collected Feedbacks (II) 14 Project # | Title of the project Sound -responsive to the movements -easy to manipulate Learning of the System -interesting -intuitive -not confusing Features -easy to differentiate between movements features -noticeable difference between using one or more limbs Equipment -comfortable enough

15. Université de Mons System Evaluation: Fluid/Rigid – Low/High Energy 15 Project # | Title of the project Fluid/Rigid movements - Low/High EnergyLow/High Energy Movements Sounds generated by the system resulted to intuitive for the users with or without training

16. Université de Mons System Evaluation: Fluid/Rigid Movements 16 Project # | Title of the project Sonification of rigid movements with low energy was significantly better recognized after the training

17. Université de Mons System Evaluation: Fluid/Rigid Movements 17 Project # | Title of the project ANOVA Sum of Squaresdf Mean SquareFSig. Rigidity Low Energy Between Groups.4171 10.015.004 Within Groups.95823.042 Total 1.37524 Rigidity High Energy Between Groups.0001.007.933 Within Groups 1.37523.060 Total 1.37524 Fluidity Low Energy Between Groups.0651 1.659.210 Within Groups.89523.039 Total.96024 Fluidity High Energy Between Groups.0101.131.720 Within Groups 1.75023.076 Total 1.76024

18. Université de Mons GAME : MOVE IN THE DARK 18 Project # | Title of the project

19. Université de Mons Conclusions 19 Project 2 | Analysis of the qualities of human movement in individual performances Q1: Is it possible for the user to understand the mapping between his movement qualities (fluid – rigid, low energy – high energy, impulsive movements) and the associated sonifications while exploring the feature space? Answer: Yes Q2: Given a particular movement feature, which is the sound that represents it the most? Answer: Future Work Q3 Is it possible for the user to distinguish the proposed movement qualities only through the auditory domain ? Answer: Yes

20. Université de Mons TEAM 20 Project 2 | Analysis of the qualities of human movement in individual performances UNIGE – CASA PAGANINI - DIBRIS KTHMAASTRICHT UNIVERSITY