1. BACS Review Meeting FCT-UC Jorge Dias 17 th – 19 th March 2008 Collège de France, Paris

2. 2 FCT-UC Key Role within BACS for M13 – M24 Contributions in  WP2: Bayesian models for sensor fusion (in collaboration with College de France).  WP5.4: Computational Laban Movement Analysis (LMA) using the Bayesian framework for Task 5.4.2 (Vision-based detection and reconstruction of human actions ( MPS)) Task 5.4.3: Multi-modal sensor integration including ego-motion  WP6: D6.2: Robotic Implementation of Gaze Control and Image Stabilization

3. 3 FCT-UC Overview: Effort & Infrastructure List of people involved in BACS (changes M1-12 to M13-24)  Paid by BACS Joerg RettPhD Student1,32 Filipe FerreiraPhD Student6,6 José PradoPhD Student7,79 Amilcar PedrosaTechnical Support 3,04 Hugo FariaTechinal Support2,66 Luis SantosPhD Student1,1 Alberto NevesPhD Student4,95 Cátia PinhoPhD Student4,4 Hadi AliakbarpourPhD Student2,64 Total 34,5  Own Resources Jorge DiasProfessor2,4 Jorge LoboProfessor2,4 J Filipe FerreiraPhD student2,4 Luis MirizolaPhD student2,4 Diego FariaPhD student2,4 Total 12 Infrastructure involved in BACS (M13 – M24)  IMPEP  Nicole platform for gesture recognition

5. 5 WP 5, Task 5.4.2 Goal Computational Laban Movement Analysis (LMA) using the Bayesian framework Problem: The research field of computational Human Movement Analysis is lacking a general underlying modeling language. How to map the features into symbols? How to model human behavior? Solution: A semantic descriptor allowing to recognize a sequence of symbols taken from an alphabet consisting of motion-entities. Benefit: Establish a set of labels for observable human behavior. The possibility to build large databases with labeled training data.

6. 6 WP 5, Task 5.4.2 Goal Computational Laban Movement Analysis (LMA) using the Bayesian framework Laban Movement Analysis: Model for human behaviour Bayesian Model: Probabilistic model to analyse human interaction

7. 7 WP 5, Task 5.4.3 Goal Multi-modal sensor integration including ego-motion Biomimetic Artificial Multimodal Perception Systems  How does the observer perceive:  his own motion (egomotion)  the 3D structure of all objects in the scene  the 3D trajectory and velocity of moving objects (independent motion)? A moving observer observes a non-static 3D scene, possibly containing several moving objects:

8. 8 WP 5, Task 5.4.3 Goal Multi-modal sensor integration including ego-motion We mainly expect to contribute in developing novel perceptual computational models which: 1.are based on the fusion of perceptual modalities of vision, audition, haptic and inertial sensing; 2.mimic as closely as possible biological multimodal perceptual fusion processes; 3.perform perceptual fusion within a Bayesian framework. and in the process to:  implement unimodal perceptual modules for Bayesian cue integration within each modality.  implement and assemble modules with several existing state-of-the-art computational models of visual, auditory, haptic and vestibular perception.

9. 9 WP 6, (D6.2) Goal Gaze control and image stabilization:  rely on fusing inertial and visual sensing modalities human and biological system also combine the two sensing modalities for the same goal.  contribution of psychophysical studies Bayesian models have been successfully used to explain psychophysical experimental findings  a robotic implementation using Bayesian inference.

10. 10 WP 5, Task 5.4.2 Achievements Computational Laban Movement Analysis (LMA) using the Bayesian framework Tracking Low Level Feature Computation 3-D Points LLF Bayesian Inference LMA labels Bayesian Inference Classified behavior Processes for online classification

11. 11 WP 5, Task 5.4.2 Results – Examples and Demos Laban descriptors Behavior hypothesis Effort.Space Effort.Time Low-level features Curvature Speed Gain Tracked positions Move- ment Space 12345678910 UUUURL00RR0

12. 12 WP 5, Task 5.4.2 Future Plans D5.17FCT-UC/Probayes:  Publication on ‘Computational Laban Movement Analysis based on Vision and 3-D Position Estimation’ T31 D5.18FCT-UC/Probayes:  Publication on ‘Bayesian Model for Computational Laban Movement Analysis’ T33 D5.20FCT-UC/Probayes:  Publication on ‘Computational Laban Movement Analysis using Multi-Camera Systems’ T36

13. 13 WP 5, Task 5.4.3 Achievements Experimental Setup: The Integrated Multimodal Perception Experimental Platform (IMPEP) current version operational and new one under construction Bayesian volumetric map (BVM), egocentric log-spherical, for multimodal perception of 3D structure and motion  An egocentric, log-spherical spatial memory map has been devised as a framework for multimodal sensor fusion, named the Bayesian Volumetric Map (BVM)  This map stores the independent probabilistic states of occupancy O C and velocity V C for each cell C in a volumetric grid with log-spherical configuration

14. 14 WP 5, Task 5.4.3 Results – Examples and Demos The Integrated Multimodal Perception Experimental Platform (IMPEP). Artificial multimodal active perception system with gaze control capabilities for image stabilization and perceptual attention with:  a stereovision setup;  a binaural setup;  a motorised head platform, with inertial sensors emulating the vestibular system. current version new version under construction (PoP FP6-IST-2004-027268)

15. 15 WP 5, Task 5.4.3 Results – Examples and Demos IMPEP/BVM Framework Overview

16. 16 WP 5, Task 5.4.3 Results – Examples and Demos Using the BVM Framework for Entropy-Based Active Exploration

17. 17 WP 5, Task 5.4.3 Future Plans D5.15FCT-UC:  Integrated multimodal perception experimental platform demo V1.0 T27 D5.16FCT-UC/INRIA/CNRS-Gren./Probayes:  Publications on Bayesian models of multimodal perception of 3D structure and motion T30 D5.21FCT-UC :  Integrated Multimodal Perception Integrated Platform demo v2.0 T36 D5.25FCT-UC/ CNRS-LPPA:  Publication on ‘Bayesian visuo-inertial gaze control’ T42

18. 18 WP 6, (D6.2) Achievements Robotic implementation of gaze control and image stabilization Simple probabilistic optical flow algorithm:  population code-type data structure storing two-dimensional pdfs on the image velocity space Du, Dv as an output.  Primarily based on Zelek's adaptation of the block matching (correlation) algorithm. Bayesian program for processing of inertial data:  Bayesian model of the human vestibular system [Laurens and Droulez(2006)]  adapted to the use of inertial sensors  estimate the current angular position and angular velocity  mimicking human vestibular perception Probabilistic Block Matching Optical Flow

19. 19 WP 6, (D6.2) Results – Examples and Demos Text We can see a strong correlation between the pan and yaw, and tilt and roll signals, since the pan&tilt is compensating the observed motion at a low sample rate. The small remaining optical flow observed shows that the controller is working to some extent, although it is not fully reliable since abrupt motions are not observable. Observed yaw and roll, pan and tilt motor control, and remaining observed optical flow.

20. 20 WP 6, (D6.2) Future Plans Our work will focus on the multimodal sensor integration within WP5, and future work will address: Implement the image stabilization alrorithm with the new robotic system. Adding the magnetic data to our Bayesian implementation, providing a more robust attitude estimation. Focus on gaze control and attention models Contact partners within BACS (CNRS-LPPA):  Following previous contacts, confront partners with our models and implementation and discuss possible parallel trials of robotic and psychophysical experiments  Going beyond the initial implementation reported in D6.2, and propose joint work and subsequent publication D5.22 T24-T42 with FCT-UC/ CNRS-LPPA:  D5.25: Publication on ‘Bayesian visuo-inertial gaze control’ T-42

21. 21 WP 5 Summary 1 Major Achievements during M13-M24  Computational Laban Movement Analysis based on Vision and 3-D Position Estimation  Experimental Setup: The Integrated Multimodal Perception Experimental Platform (IMPEP) current version operational and new one under construction  Bayesian volumetric map (BVM), egocentric log-spherical, for multimodal perception of 3D structure and motion List of Deliverables  D5.7 (MPS) Report on computational human pose recovery in clutter M18  D5.8 (FCT-UC) State of the art on (Artificial) 3D Structure and Motion Multimodal Perception M15  D6.2 (FCT-UC) Robotic Implementation of Gaze Control and Image Stabilization M18

22. 22 WP 5 Summary 1 Conference  Rett, J., Dias, J.: Human-robot interface with anticipatory characteristics based on Laban Movement Analysis and Bayesian models. In: Proceedings of the IEEE 10th International Conference on Rehabilitation Robotics (ICORR). (2007)  Rett, J., Dias, J.: Human Robot Interaction based on Bayesian Analysis of Human Movements. In: Proceedings of EPIA 07, Lecture Notes in AI, Springer Verlag, Berlin. (2007)  Luiz G. B. Mirisola, Jorge Lobo, and Jorge Dias. 3D map registration using vision/laser and inertial sensing. In European Conference on Mobile Robots (ECMR2007), Freiburg, Germany, Sep. 2007.  J. Dias, Carlos Simplicio, Diego R. Faria - 3D Photo-realistic talking head for human-robot interaction - Proceedings of the 3rd Internacional Conference on Advanced Research in Virtual and Rapid Prototyping, Leiria, Portugal, 24 to 29 September, 2007  F. Ferreira, V. Santos and Jorge Dias, Robust Place Recognition Within Multi-sensor View Sequences Using Bernoulli Mixture Models, The 6th IFAC Symposium on Intelligent Autonomous Vehicles, IAV 2007, Toulouse, France  Rett, J. and Dias, J. “Computational Laban Movement Analysis using probability calculus.” In the Proceedings of Workshop on Robotics and Mathematics, RoboMat 2007.  Rett, J. and Dias, J.:”Bayesian models for Laban Movement Analysis used in Human Machine Interaction.” Proceedings of ICRA 2007 Workshop on "Concept Learning for Embodied Agents“.  Ferreira, F., Davim, L., Rocha, R., Santos, V. and Dias, J.:”Using Local Features To Classify Objects Having Printable Codes”. Proceedings of the International Conference 5th Workshop on European Scientific and Industrial Collaboration on promoting Advanced Technologies in Manufacturing, WESIC Journal  Jorge Lobo and Jorge Dias, "Relative Pose Calibration Between Visual and Inertial Sensors", International Journal of Robotics Research, Special Issue 2nd Workshop on Integration of Vision and Inertial Sensors, vol.26, n.6, June 2007, pages 561-575.  Peter Corke, Jorge Lobo and Jorge Dias, "An introduction to inertial and visual sensing", International Journal of Robotics Research, Special Issue 2nd Workshop on Integration of Vision and Inertial Sensors, vol.26, n.6, June 2007, pages 519-535. Thesis  Jorge Lobo, "Integraton of Vision and Inertial Sensing", PhD Thesis, Supervisor: Jorge Dias, defended July 2007.

23. 23 WP 5 Summary 2 Major collaborations within BACS  RDG2: collaboration between FCT-UC, IDIAP, ETH Zürich and MPS Tübingen was initiated, with the purpose of developing tools for audiovisual VR world production, so as to create stimuli both for human perception studies (WP6.4) and for simulations related to this Task, so as to test and demo the artificial multimodal perception system. This ongoing effort has already produced a co-authored State-of-the-Art Report: “State of the Art on 3D Audiovisual APIs/SDKs for Stimulus Generation and Presentation”.  RDG3, a collaboration effort between the FCT-UC, INRIA Rhône-Alpes, Probayes and CNRS-Grenoble concerning the subject “Bayesian Models for Multimodal Perception of 3D Structure and Motion” involving a student exchange from M20 to M22 was undertaken, having been successfully completed and for which several deliverables are being finalised and expected to be ready by M24, namely: A final Technical Report, describing the work done during the exchange period and delineating future collaboration between these partners. 2 joint-publications, to be submitted for ICVS 2008 and CogSys 2008 … Major collaborations outside BACS  FCT-UC collaborates with the Perception on Purpose project (PoP FP6-IST-2004- 027268, of which FCT-UC is also a partner) sharing common physical resources and know-how, namely in the construction of the new robotic vision head Summary of future plans  Computational Laban Movement Analysis using Multi-Camera Systems  Bayesian models of multimodal perception of 3D structure and motion  Going beyond the initial implementation reported in D6.2, and propose joint work and subsequent publication D5.22 T24-T42 with FCT-UC/ CNRS-LPPA: