Visual Event Detection & Recognition Filiz Bunyak Ersoy, Ph.D. student Smart Engineering Systems Lab

Research Interest Recognition of visual events from video sequences. Recognition of visual events from video sequences. Use of visual event information for intelligent video surveillance and semantic video indexing & retrieval. Use of visual event information for intelligent video surveillance and semantic video indexing & retrieval. Incorporation of learning to event modeling and recognition. Incorporation of learning to event modeling and recognition. Very few event detection systems that are currently available are not flexible, work for a very limited domain for very limited number of predefined, mostly hand-coded events. They are not designed to be extended or modified. Learning will enable adaptability and extensibility of an event detection system.

Content of Video Data Low Level Visual Features Color Color Texture Texture Shape Shape Motion Motion Shot Boundaries Shot Boundaries Mid Level Semantic Content People/Objects Location Actions Time High Level Semantic Content Story Story Concept Concept Event Event

Challenges1-Representation: Modeling visual features Modeling visual features Image/video object representation Image/video object representation Description of spatio- temporal relationships Description of spatio- temporal relationships High-level event representation High-level event representation2-Analysis: Segmentation of video objects Segmentation of video objects Adaptive grouping of features & objects Adaptive grouping of features & objects Compressed-domain feature extraction Compressed-domain feature extraction 3-Indexing: Efficient indexing algorithms for high-dimensional feature space Robust, scalable indexing algorithms for spatio-temporal queries 4-Summarization: Automated summarization of visual content Visualization of content at different levels

Proposed Framework Feature Extraction Motion Analysis Event Inference Object Classification Objects Relationships Events Context Object, Scene & Event Libraries Context is any a priori information provided to the system. Events Object Information Object Trajectories & Spatio-temporal relationships

Motion Analysis Moving Object Detection Temporal differencing Temporal differencing Background subtraction Background subtraction Optical Flow Optical Flow Tracking Region-based methods Contour-based methods Feature-based methods Model-based methods Moving Object Detection Feature Extraction Correspondence Analysis Prediction Update Context Object States Tracking Single-view methods (single camera) Single-view methods (single camera) Multi-view methods (multiple cameras) Multi-view methods (multiple cameras)

Event Inference Possible Event Inference Methods Rule based Rule based Logical Formalisms Logical Formalisms Temporal Logic Temporal Logic Event Logic Event Logic Fuzzy Logic Fuzzy Logic Bayesian belief network Bayesian belief network Hidden Markov model Hidden Markov model Petri-net Petri-net Grammar Grammar Event Inference Methods Information About Objects Spatio-temporal Relationships A priori Information About the Application, Goal, Scene, Objects. Event descriptions. EVENTS

Applications of Event Detection/Recognition Surveillance and Monitoring Surveillance and Monitoring Traffic (track vehicle movements and annotate action in traffic scenarios.) Traffic (track vehicle movements and annotate action in traffic scenarios.) Detection of accidents, traffic violations, congestions. Detection of accidents, traffic violations, congestions. Gather statistics about human activities, road utilization etc. Gather statistics about human activities, road utilization etc. Surveillance of public places / shops / offices etc. Surveillance of public places / shops / offices etc. Detection of atypical incidents, theft, vandalism, shoplifting, abandoning (possibly dangerous) objects. Detection of atypical incidents, theft, vandalism, shoplifting, abandoning (possibly dangerous) objects. Indexing of Broadcast Video Sports video indexing for newscasters or trainers. Semantic indexing for automated annotation for content retrieval. Interactive environments: environment that respond to the activity of occupants. Robotic collaboration: creating robots that can effectively navigate their environment and interact with other people and robots.

Some Interesting Systems from the Industry-1 Realtime Video Analysis Siemens Corporate Research Subway monitoring System Real time segmentation of people in subway platforms for the purpose of congestion (crowding) detection Real time segmentation of people in subway platforms for the purpose of congestion (crowding) detection Honeywell Laboratories: Cooperative Camera Network (CCN): Indoor Reports the presence of visually tagged individual throughout a building structure. Meant to be used for monitoring potential shoplifters in department stores. Detection of Events for Threat Evaluation & Recognition (DETER): Outdoor Monitor large open spaces like parking lots and reports unusual moving patterns by pedestrians & vehicles.

Some Interesting Systems from the Industry-2 Mitsubishi Electric Research Laboratories Applications Detecting accidents through analysis of traffic surveillance video. Detecting accidents through analysis of traffic surveillance video. Detection of traffic jams using our MPEG-7 motion activity descriptor. Detection of traffic jams using our MPEG-7 motion activity descriptor. Extraction of semantic features from low- level features of soccer games. Extraction of semantic features from low- level features of soccer games.

Some Interesting Systems from the Industry-3 ASCOM: INVIS Traffic Detect Traffic speed and traffic flow density Traffic speed and traffic flow density Congestion and slow traffic Congestion and slow traffic Stationary vehicles (possibly an accident) Stationary vehicles (possibly an accident) Copyright © 2002 Ascom Wrong Way Early Warning System Recognizes vehicle patterns and compares subsequent images to determine vehicle and direction. Thus it can detect any car driving the wrong way into a lane against the flow.