Research and Future Perspectives on Intelligent Video Surveillance Systems Monique THONNAT Senior Scientist Head of Orion research team INRIA Sophia Antipolis FRANCE

2 Introduction 1/4 Which Security Problems? Safety and security of goods and human beings  Safety: protection in case of accident or incident (e.g.: fall)  Security: protection against a malicious act (ex.: bomb) Evolution  1980: guarding, human surveillance  1990: start of video cameras setting up, remote surveillance  Highways, parking lots, subways, malls …  2000: explosion of video-surveillance  Set of new laws  Better understanding of the general public  Threats of tragic events, terrorism acts

3 Introduction: control room 2/4

4 Introduction: issue 3/4 Definition: intelligent video surveillance automatic analysis of the video streams Why ?  Cost: 1 human operator /6-10 video streams  Current paradox : the more there are video cameras, the less these video camera are observed  E.g video cameras in London  need for the policemen to look at stored videos tapes  Effectiveness: duration of vigilance for an operator:  > 1 or 2hours: % of the attention is lost

6 Intelligent Video Surveillance 1/3 demonstration From object detection to complex event recognition (e.g.: violence)

7 Intelligent Video Surveillance Definition: Data captured by video surveillance cameras Real time and automated analysis of video sequences Video understanding= from people detection and tracking to behavior recognition Recognition of complex behaviors: of individuals (e.g. fraud, graffiti, vandalism, bank attack) of small groups (e.g. fighting) of crowds (e.g. overcrowding) interactions of people and vehicles (e.g. aircraft refueling) Intelligent Video Surveillance 2/3

8 Intelligent Video Surveillance 3/3 Typical problems Metro station surveillance Surveillance inside trains Building access controlAirport monitoring

9 Video Understanding for Intelligent Video Surveillance 1/13 Definition: Cognitive Vision is a new research field mixing: Computer vision techniques for object detection, description, categorization and tracking Artificial intelligence techniques for knowledge acquisition, reasoning (e.g. spatial and temporal reasoning,…), learning (e.g. categories, structures, parameters…) Software engineering techniques for vision software design, integration, reusability, evaluation Reference: site and roadmap

10 Intelligent Videosurveillance: How? A Cognitive vision approach for video understanding mixing:  computer vision: 4D analysis (3D + temporal analysis)  artificial intelligence: a priori knowledge (scenario, environment)  software engineering: reusable software platform (VSIP) Video Understanding for Intelligent Video Surveillance 2/13

11 Video Understanding for Intelligent Video Surveillance 3/13 4 D analysis: multi-cameras tracking Video understanding People detection and tracking Scenario recognition A PRIORI KNOWLEDGE: 3d models of the environment Camera calibration Scenario Models Alarms People detection and tracking Interpretation of the videos from pixels to alarms

12 Video Understanding for Intelligent Video Surveillance 4/13 Segmentation Classification Tracking Scenario Recognition Alarm s access to forbidden area 3D scene model Scenario models A priori Knowledge Objective: Interpretation of videos from pixels to alarms

13 Behavior recognition:  approach based on a priori knowledge  model of the empty scene (3D geometry and semantics)  models of predefined scenarios  a language for representing scenarios based on combination of states and events  more than 20 states and 20 events can be used  a reasoning mechanism for real time detection of states, events and scenarios (e.g. temporal reasoning, constraints solving techniques) Video Understanding for Intelligent Video Surveillance 5/13

14 3D Scene Model: Barcelona Metro Station Sagrada Famiglia mezzanine (European project ADVISOR) Video Understanding for Intelligent Video Surveillance 6/13

15 States, Events and Scenarios : State: a spatio-temporal property involving one or several actors on a time interval Ex : « close», « walking», « seated» Event: a significant change of states Ex : « enters», « stands up», « leaves » Scenario: a long term symbolic application dependent activity Ex : « fighting», « vandalism» Video Understanding for Intelligent Video Surveillance 7/13

16 Vandalism scenario description : Scenario(vandalism_against_ticket_machine, Physical_objects((p : Person), (eq : Equipment, Name = “Ticket_Machine”) ) Components ((event s1: p moves_close_to eq) (state s2: p stays_at eq) (event s3: p moves_away_from eq) (event s4: p moves_close_to eq) (state s5: p stays_at eq) ) Constraints ((s1 != s4) (s2 != s5) (s1 before s2) (s2 before s3) (s3 before s4) (s4 before s5) ) ) ) Video Understanding for Intelligent Video Surveillance 8/13 Scenario Recognition : Temporal constraints

17 Video Understanding for Intelligent Video Surveillance 9/13 Vandalism in metro (Nuremberg, Germany)

18 Video Understanding for Intelligent Video Surveillance 10/13 3d Scene Model of 2 bank agencies objet du contexte mur et porte zone d’accès salle du coffre ru e salle automates zone d’entrée de l’agence zone des distributeurs zone de jour/nuit zone devant le guichet zone derrière le guichet zone d’accès au bureau du directeur zone de jour porte d’entrée porte salle automates armoire guichet commode

19 Video Understanding for Intelligent Video Surveillance 11/13 Bank Monitoring: Bank attack scenario description : scenario Bank_attack_one_robber_one_employee physical_objects: ((employee : Person), (robber : Person), z1: Back_Counter, z2: Entrance_Zone, z3: Front_Counter, z4: Safe, d: Safe_door) components: (State c1 : Inside_zone(employee, z1)) (Event c2 : Changes_zone(robber, z2,z3)) (State c3 : Inside_zone(employee, z4)) (State c4 : Inside_zone(robber, z4))) constraints : ((c2 during c1) (c2 before c3) (c1 before c3) (c2 before c4) (c4 during c3) (d is open))

20 Video Understanding for Intelligent Video Surveillance 12/13 bank monitoring Recognition of a bank attack scenario: an employee is e behind a counter, an aggressor enters, goes behind the counter, then he goes with the emplyee towards the STR (secured technical room), they enter in the STR then they leave ethe STR and they go toward the exit of the agency.

Examples : Brussels and Barcelona Metro Surveillance Exit zone Jumping over barrier Blocking Overcrowdin g Fighting Group behavior Crowd behavior Individual behavior Group behavio r Video Understanding for Intelligent Video Surveillance 13/13 21

22 Impact: Visual surveillance of metro stations, bank agencies, trains, buildings and airports  5 European projects (PASSWORDS, AVS-PV, AVS-RTPW, ADVISOR, AVITRACK)  4 contracts with End-users companies (metro, bank, trains)  2 transfer activities with Bull (Paris) and Vigitec (Brussels) Cooperation over more than 11 years with partners Creation in 2005 of a start-up Keeneo Conclusion 1/5

23 Conclusion 2/5 Hypotheses:  fixed cameras  3D model of the empty scene  predefined behavior models Results: + Behavior understanding for Individuals, Groups of people, Crowd or Vehicles + an operational language for video understanding (more than 20 states and events) + a real-time platform (10 to 25 frames/s)

24 Conclusion 3/5 Current issues  Systems have poor performances over time, can be hardly modified and do not use enough a priori knowledge shadows strong perspective tiny objects close view clutte r lighting condition s

25 Knowledge Acquisition  Design of learning techniques to complement a priori knowledge:  Frequent events, scenario model learning  European project CARETAKER Object description  Fine human shape description : 3D posture models  Crowd description: European project SERKET Reusability is still an issue for vision programs  Video analysis robustness  Dynamic configuration of programs and parameters Conclusion: Where we go 4/5

26 Conclusion 5/5 Posture Recognition Current image and binary image Instantaneous posture Postures recognised along the time

27 Conclusion 5/5 Crowd Behavior Motion direction detection Abnormal direction of people in a crowd

28 Video Understanding demo? Airport Apron Monitoring “Unloading Operation”“Unloading Operation” European AVITRACK project