Tracking-dependent and interactive video projection (Big Brother project)

Team Pierre Bretéché Matei Mancas Jonathan Demeyer Thierry Ravet Donald Glowinski Gualtiero Volpe

System Architecture

Summary Robust tracking Modeling human motion attention of a visual scene

Summary Robust tracking Tracking should be robust to severe illumination changes ! Modeling human motion attention of a visual scene

Robust tracking: data acquisition Color tracking of red hats IR tracking of lights on the top of the red hats

Robust tracking: data fusion Projective transform

Robust tracking: data fusion Selected points (Matlab) Transformation matrix (EyesWeb) Image warping (EyesWeb)

Robust tracking: data fusion RED: color tracking / GREEN : IR tracking / BLUE : Final fused tracking

Summary Robust tracking Modeling human motion attention of a visual scene

Summary Robust tracking Modeling human motion attention of a visual scene Regions of interest should be dynamically highlighted and visual effects corresponding to outstanding events should be displayed

Attention: in space (blob speed) HOT RED : most important / DARK RED : less important Global contrast !

Attention: in time (blob speed) HOT RED : really interesting / BLACK : really boring Rarity on 4 seconds

Attention: in time (quantity of motion) HOT RED : really interesting / BLACK : really boring Rarity on 4 seconds

Conclusion Summary : Multi-blob real time tracking (on IR images and color images) Image registration using projective warping Data fusion using a weighted combination of blob positions based on confidence levels for each modality Demonstration of motion attention in space (instantaneous) and in time (short- time memory) -> motion is not necessarily salient : it depends on the context Use of blob speed and silhouette quantity of motion features So we worked hard...

Conclusion... but we will work even harder : Refinement of confidence level for each modality Better hardware is needed: adapted optics for the cameras, and a color camera with the same characteristics than the IR camera: easy modality registration A computer with several firewire ports: easy modality synchronization Use of other features for attention for tracked paths (direction, acceleration, direction variation, trajectory curvature, …) and for gesture expressivity (energy, internal/external quantity of motion, …) Long-time memory attention (flickering lights, commonly used paths, …) to get higher level motion segmentation More efficient instantaneous attention (space) which works also with moving cameras and for surprising behaviors in crowds …

Thank you !