University of Coimbra ISR – Institute of Systems and Robotics University of Coimbra - Portugal WP5: Behavior Learning And Recognition Structure For Multi-modal Fusion Part I

University of Coimbra Relationship of the WP3,4 and 5 WP3 (Sensor modeling and multi-sensor fusion techniques ) Task 3.3 Bayesian network structures for multi-modal fusion WP4 (Localization and tracking techniques as applied to humans ) Task 4.3 Online adaptation and learning WP5 Behavior learning and recognition Trackers results, Events detected, ids on re-identification situations Levels of the fusion (pixel, feature or decision level), Bayesian structures for the implementation of the scenarios of WP2

University of Coimbra Proposal Multi-layer Multi-modal Homography-based Occupancy Grid Using data coming from stationary sensors (Structure): – Image Data – Range Data – Sound Source Data

University of Coimbra Inertial Compensated Homography Projecting a world point on a reference plane in two phases X Y Z Real camera Virtual camera [Luiz2007] Gravity First step: Projecting real world point on the virtual image plane Second step: Projecting form virtual image plane on a common plane

University of Coimbra Inertial Compensated Homography X Y Z Real camera Virtual camera [Luiz2007] Gravity Infinite homography Homography between two planes Camera calibration matrix Rotation between virtual and real camera (given by IMU)

University of Coimbra Image Registration X Y Z [Luiz2007] Gravity

University of Coimbra Extending A Virtual Plane To More Plane to image homography: Vanishing points for X,Y and Z directions Vanishing line of reference plane normalized [Khan2007] Vanishing point of reference direction Scale factor Scale value encapsulating  and z X Y Z

University of Coimbra Relationship Between Different Planes In The Structure Homography between views i and j, induced by a plane  i parallel to  ref having the homography of reference plane: [Khan2007] X Y Z Vanishing point of reference direction Scale value Homography of reference plane

University of Coimbra Image & Laser Geometric Registration X Y Z [Luiz2007-Hadi2009] Gravity

University of Coimbra Registering LRF Data In a Multi-Camera Scenario [Hadi2009] Reprojection of LRF data on the image (blue points) Image planes Projection of points observed by LRF Transformation matrix between camera and LRF obtained by calibration Projection of points observed by LRF on the image plane + Result A set of cameras and laser range finder Camera projection matrix Image Range data

University of Coimbra Image & Laser & Sound Geometric Registration X Y Z Gravity PA(  ) [JFC2008, 2009]

University of Coimbra Bayesian Binaural System for 3D Localisation –Binaural sensing interaural time and level differences – ITD , quasi frequency-independent, and ILDs  L(f c k ))For sources within 2 meters range, binaural cues alone (interaural time and level differences – ITD , quasi frequency-independent, and ILDs  L(f c k )) can be used to fully localise the source in 3D space (i.e. volume confined in azimuth , elevation  and distance  ). If the source is more than 2 meters away the source can only be localised to a volume (cone of confusion) in azimuth. 1m2m Binaural cue information + + Distance  Elevation  Azimuth θ Azimuth θ only Z

University of Coimbra Bayesian Binaural System for 3D Localisation Subset of [JFC2008, 2009]

University of Coimbra Direct Auditory Sensor Model: () Direct Auditory Sensor Model: (DASM) (Bayesian learning through HRTF calibration using ITDs  and ILDs  L) Azimuth Elevation Distance Binary variable denoting “Cell C occupied by sound-source” Inverse Auditory Sensor Model: () Inverse Auditory Sensor Model: (IASM) Bayesian Binaural System for 3D Localisation Bayes Rule Auditory Saliency Map Solution: cluster local saliency maxima points (i.e. cells with maximum probability of occupancy, 1 per sound-source) (front -to-back confusion effect avoided by considering only frontal hemisphere estimates)

University of Coimbra Bayesian Binaural System for Localisation in Azimuth Planes of Arrival Direct Auditory Sensor Model: () Direct Auditory Sensor Model: (DASM) (Bayesian learning through HRTF calibration of interaural time differences – ITDs  ) Inverse Auditory Sensor Model: () Inverse Auditory Sensor Model: (IASM)  -90º90º Bayes Rule Auditory Saliency Map Solution: cluster local saliency maxima planes of arrival (PA) per sound-source PA(  ) (front -to-back confusion effect avoided by considering only frontal hemisphere estimates)

University of Coimbra Demos on Bayesian Binaural System (Arrival Direction of Sound Source) Two Talking personsA walking person

University of Coimbra A Sample View Of.

University of Coimbra Image & Laser & Sound Occupancy Grid Image, Range and Sound Occupancy Grid X Z Y X Fusion Y Z

University of Coimbra Bibliography

University of Coimbra Bibliography Franco, J. & Boyer, E. Fusion of Multi-View Silhouette Cues Using a Space Occupancy Grid Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV’05), 2005 Christophe Braillon, Kane Usher, C. P. J. L. C. & Laugier, C. Fusion of stereo and optical flow data using occupancy grids Computer Vision and Pattern Recognition, CVPR IEEE Conference on, Saad M. Khan, P. Y. & Shah, M. A Homographic Framework for the Fusion of Multi-view Silhouettes Computer Vision, ICCV IEEE 11th International Conference on, 2007 R. Eshel, Y. M. Homography Based Multiple Camera Detection and Tracking of People in a Dense Crowd Computer Vision and Pattern Recognition, CVPR IEEE Conference on, 2008 Conference (Arsic2008) D. Arsic, E. Hristov, N. L. Applying multi layer homography for multi camera person tracking Distributed Smart Cameras, ICDSC Second ACM/IEEE International Conference on, 2008 Francois Fleuret, Jerome Berclaz, R. L. & Fua, P. Multi-Camera People Tracking with a Probabilistic Occupancy Map IEEE transactions on Pattern analysis and Machine Intelligence, 2008

University of Coimbra Bibliography Sangho Park, M. M. T. Understanding human interactions with track and body synergies (TBS) captured from multiple views Computer Vision and Image Understanding, 2008 Yuxin Jin, Linmi Tao, H. D. R. N. & Xu, G. Background modeling from a free-moving camera by Multi-Layer Homography Algorithm Image Processing, ICIP th IEEE International Conference on, 2008 Luiz G. B. Mirisola, Jorge Dias, A. T. d. A. Trajectory Recovery and 3D Mapping from Rotation- Compensated Imagery for an Airship Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems San Diego, CA, USA, Oct 29 - Nov 2, 2007, 2007 Mirisola, L. G. B. & Dias, J. Tracking from a Moving Camera with Attitude Estimates ICR08, 2008 Batista, J. P. Tracking Pedestrians Under Occlusion Using Multiple Cameras Image Analysis and Recognition, Springer Berlin-Heidelberg., 2004, 3212/2004, Joao Filipe Ferreira, Pierre Bessière, K. M. C. P. J. L. C. L. & Dias, J. Bayesian Models for Multimodal Perception of 3D Structure and Motion C. Chen, C. Tay, K. M. & C. Laugier (INRIA, F. Dynamic environment modeling with gridmap: a multiple-object tracking application 9th International Conference on Control, Automation, Robotics and Vision, ICARCV '06., 2006

University of Coimbra Bibliography J. F. Ferreira, P. Bessière, K. Mekhnacha, J. Lobo, J. Dias, and C. Laugier, “Bayesian Models for Multimodal Perception of 3D Structure and Motion,” in International Conference on Cognitive Systems (CogSys 2008), pp , University of Karlsruhe, Karlsruhe, Germany, April C. Pinho, J. F. Ferreira, P. Bessière, and J. Dias, “A Bayesian Binaural System for 3D Sound- Source Localisation,” in International Conference on Cognitive Systems (CogSys 2008), pp , University of Karlsruhe, Karlsruhe, Germany, April Ferreira, J. F., Pinho, C., and Dias, J., “Implementation and Calibration of a Bayesian Binaural System for 3D Localisation”, in 2008 IEEE International Conference on Robotics and Biomimetics (ROBIO 2008), Bangkok, Tailand, Hadi Aliakbarpour, Pedro Nunez, Jose Prado, Kamrad Khoshhal and Jorge Dias. An Efficient Algorithm for Extrinsic Calibration between a 3D Laser Range Finder and a Stereo Camera for Surveillance, ICAR2009.

University of Coimbra Institute of Systems and Robitcs