Project on Visual Monitoring of Human Behavior and Recognition of Human Behavior in Metro Stations with Video Understanding M. Thonnat Projet ORION INRIA Sophia Antipolis, France 18/03/2003 Inria/NSC

2 Project on Visual Monitoring of Human Behavior Recognition of Human Behavior in Metro Stations with Video Understanding Video Understanding Results for Metro Stations Conclusion Outline

18/03/2003 Inria/NSC 3 Cooperation Project Title: Visual Monitoring of Human Behavior Involved teams : 1- Yi-Ping Hung & Chu-Song Chen, Academia Sinica,Taipei,Taiwan 2- Pau-Choo Chung, National Cheng Kung Univ., Tainan,Taiwan 3- Monique Thonnat, INRIA Sophia Antipolis, France Objectives: Sub-goal 1: Human Detection, Tracking and Recognition using Video Cameras (mainly teams 1 and 3) Sub-goal 2: Behavior Recognition for Medical Purposes (mainly teams 2 and 3)

Recognition of Human Behavior in Metro Stations with Video Understanding M. Thonnat, F. Cupillard, F. Bremond and A. Avanzi Projet ORION INRIA Sophia Antipolis, France 18/03/2003 Inria/NSC

5 Objective: to automate the recognition of specific human behaviors from video sequences ---> Interpretation of the videos from pixels to alarms Video understanding « Blocking an exit » «Fighting » « Fraud » « Overcrowding» ALARM Interface for alarm management Behavior recognition People detection and tracking Cameras

18/03/2003 Inria/NSC 6 Video Understanding Context: European project ADVISOR: Annotated Digital Video for Intelligent Surveillance and Optimised Retrieval Intelligent system of video surveillance in metros Problem : 1000 cameras but few human operators Automatic selection in real time of the cameras viewing abnormal behaviours Automatic annotation of recognised behaviors in a video data base using XML

18/03/2003 Inria/NSC 7 Scene Models (3D) - Scene objects - zones - calibration matrices - Scene objects - zones - calibration matrices Alarms Multi-cameras Combination Behaviors Recognition - States - Events - Scenarios Individual Tracking Group Tracking Crowd Tracking - Motion Detector - F2F Tracker - Motion Detector - F2F Tracker - Motion Detector - F2F Tracker Mobile objects Annotation Scenario Models Video Understanding Platform

18/03/2003 Inria/NSC 8 Definition : a priori knowledge of the observed empty scene Cameras: 3D position of the sensor, calibration matrix field of view,... 3D Geometry of physical objects (bench, trash, door, walls) and interesting zones (entrance zone) with position, shape and volume n Semantic information : type (object, zone), characteristics (yellow, fragile) and its function (seat) Role: to keep the interpretation independent from the sensors and the sites : many sensors, one 3D referential to provide additional knowledge for behavior recognition Video Understanding: 3D Scene Model

18/03/2003 Inria/NSC 9 Barcelona Metro Station Sagrada Famiglia mezzanine (cameras C10, C11 and C12) Video Understanding: Scene Model

18/03/2003 Inria/NSC 10 States, Events and Scenarios : State: a spatio-temporal property involving one or several actors on a time interval Ex : « close», « walking», « seated» Event: a significant change of states Ex : « enters», « stands up», « leaves » Scenario: a long term symbolic application dependent activity Ex : « fighting», « vandalism» Video Understanding

18/03/2003 Inria/NSC 11 Several types of States : posture  {lying, crouching, standing} direction  {towards the right, towards the left, leaving, arriving} speed  {stopped,walking, running} distance/object  {close, far} distance/person  {close, far} posture/object  {seated, any} Several types of Events : 1 person : falls down, crouches down, stands up, goes right side, goes left side, goes away, arrives, stops, starts running 1 person & 1 zone : leaves, enters 1 person & 1 equipment : moves close to,sits on, moves away from 2 persons : moves close to, moves away from Video Understanding

18/03/2003 Inria/NSC 12 We use several formalisms to recognise states, events and scenarios : specific routines classification finite state automaton propagation of temporal constraints Scenario Recognition

Exit zone Mobile objects Detection Group Tracking Recognition of the behaviour « A Group of people blocks an Exit» Blocking The operator of the scenario “A Group of people blocks an Exit” is based on a Finite state automaton Scenario Recognition : Automaton INIT Grp x is tracked Grp x is inside a ZOI Grp X is stopped in the ZOI > 30 sec Enter_ZOI Exit_ZOI « Blocking » Stops Start_walking Start_running 13 26/02/2003 IDSS

18/03/2003 Inria/NSC 14 A temporal scenario is constituted by three parts : Characters : people, physical objects and interesting zones relative to the scenario. Constraints : a set of constraints on the characteristics of the actors and on the states, events and sub-scenarios involving the actors. Production : generation of a scenario instance which can be part of more complex scenarios. Use of symbolic, logical, spatial and temporal constraints including Allen’s interval algebra operators. Scenario Recognition : Temporal constraints

18/03/2003 Inria/NSC 15 Vandalism scenario description : Scenario(vandalism_against_ticket_machine, Characters((p : Person), (eq : Equipment, Name = “Ticket_Machine”) ) Constraints((exist ((event s1: p move_close_to eq) (state s2: p stay_at eq) (event s3: p move_away_from eq) (event s4: p move_close_to eq) (state s5: p stay_at eq) ) ((s1 != s4) (s2 != s5) (s1 before s2) (s2 before s3) (s3 before s4) (s4 before s5) ) ) ) Production( (sc : Scenario) ( (Name of sc := "vandalism_against_ticket_machine") (StartTime of sc := StartTime of s1) (EndTime of sc := EndTime of s5) ) ) ) Scenario Recognition : Temporal constraints

18/03/2003 Inria/NSC 16 Results Vandalism in metro (Nuremberg)

Examples : Brussels and Barcelona Metros Exit zone Jumping over barrier Blocking Overcrowding Fighting Group behavior Crowd behavior Individual behavior Group behavior Results 17 26/02/2003 IDSS

18/03/2003 Inria/NSC 18 Recognition of five behaviors: “Blocking”, “Fighting”, “Jumping over barrier”, “Vandalism” and “Overcrowding” Tested on 50 metro sequences (10 hours) True positive per sequence: 70% (“Fighting”) to 95% (“Blocking”) False positive per sequence: 5% (“Fighting”, “Jumping over barrier”) to 0% (others) Results

18/03/2003 Inria/NSC 19 Conclusion Hypotheses: fixed cameras 3D model of the empty scene predefined behavior models Results: Behavior understanding for Individuals, Groups of people or Crowd an operational language for video understanding (more than 20 states and events) a real-time platform (5 to 25 frames/s)

18/03/2003 Inria/NSC 20 Future work: Starting live evaluation at Barcelona Metro Learning techniques to compute optimal set of operator parameters and to dynamically configure the platform Visual Monitoring of Human Behavior project Conclusion