Data Mining for Surveillance Applications Suspicious Event Detection Dr. Bhavani Thuraisingham November 2007

Problems Addressed Huge amounts of video data available in the security domain Analysis is being done off-line usually using “Human Eyes” Need for tools to aid human analyst ( pointing out areas in video where unusual activity occurs) Consider corporate security for a fenced section of sensitive property The guard suspects there may have been a breach of the perimeter fence at some point during the last 48 hours They must: Manually review 48 hours of tape Consider multiple cameras and camera angles Distinguish between normal personnel and intruders

Example Using our proposed system: Greatly Increase video analysis efficiency User Defined Event of interest Video Data Annotated Video w/ events of interest highlighted

The Semantic Gap The disconnect between the low-level features a machine sees when a video is input into it and the high-level semantic concepts (or events) a human being sees when looking at a video clip Low-Level features: color, texture, shape High-level semantic concepts: presentation, newscast, boxing match

Our Approach Event Representation Event Comparison Event Detection Estimate distribution of pixel intensity change Event Comparison Contrast the event representation of different video sequences to determine if they contain similar semantic event content. Event Detection Using manually labeled training video sequences to classify unlabeled video sequences

Event Representation Measures the quantity and type of changes occurring within a scene A video event is represented as a set of x, y and t intensity gradient histograms over several temporal scales. Histograms are normalized and smoothed

Event Comparison Determine if the two video sequences contain similar high-level semantic concepts (events). Produces a number that indicates how close the two compared events are to one another. The lower this number is the closer the two events are.

Event Detection A robust event detection system should be able to Recognize an event with reduced sensitivity to actor (e.g. clothing or skin tone) or background lighting variation. Segment an unlabeled video containing multiple events into event specific segments

Walking1 Running1 Waving2 Labeled Video Events These events are manually labeled and used to classify unknown events Walking1 Running1 Waving2

Labeled Video Events walking1 walking2 walking3 running1 running2   walking1 walking2 walking3 running1 running2 running3 running4 waving 2 0.27625 0.24508 1.2262 1.383 0.97472 1.3791 10.961 0.17888 1.4757 1.5003 1.2908 1.541 10.581 1.1298 1.0933 0.88604 1.1221 10.231 0.43829 0.30451 0.39823 14.469 0.23804 0.10761 15.05 0.20489 14.2 15.607 waving2

Experiment #1 Problem: Recognize and classify events irrespective of direction (right-to-left, left-to-right) and with reduced sensitivity to spatial variations (Clothing) “Disguised Events”- Events similar to testing data except subject is dressed differently Compare Classification to “Truth” (Manual Labeling)

Classification: Walking Experiment #1 Disguised Walking 1 walking1 walking2 walking3 running1 running2 running3 running4 waving2 0.97653 0.45154 0.59608 1.5476 1.4633 1.5724 1.5406 12.225 Classification: Walking

Classification: Walking Experiment #1 Disguised Walking 2 walking1 walking2 walking3 running1 running2 running3 running4 waving2 0.948 0.38097 0.53852 1.9412 1.844 1.8711 1.9673 10.191 Classification: Walking

Classification: Running Experiment #1 Disguised Running 1 walking1 walking2 walking3 running1 running2 running3 running4 waving2 1.411 1.3841 1.0637 0.56724 0.97417 0.93587 1.0957 11.629 Classification: Running

Classification: Running Experiment #1 Disguised Running 2 walking1 walking2 walking3 running1 running2 running3 running4 waving2 1.3543 1.1909 1.0071 0.61541 0.95833 0.94227 0.93731 13.141 Classification: Running

Classifying Disguised Events Disguised Running 3 walking1 walking2 walking3 running1 running2 running3 running4 waving2 1.3049 1.0021 0.88092 0.8114 1.1042 1.1189 1.0902 12.801 Classification: Running

Classifying Disguised Events Disguised Waving 1 walking1 walking2 walking3 running1 running2 running3 running4 waving2 13.646 13.113 13.452 18.615 19.592 18.621 20.239 2.2451 Classification: Waving

Classifying Disguised Events Disguised Waving 2 walking1 walking2 walking3 running1 running2 running3 running4 waving2 12.792 12.132 12.681 18.104 18.956 18.029 19.547 3.1336 Classification: Waving

Classifying Disguised Events   Disguise walking1 walking2 running1 running2 running3 waving1 waving2 0.19339 1.2159 0.85938 0.67577 14.471 13.429 1.4317 1.1824 0.95582 12.295 11.29 0.37592 0.45187 15.266 15.007 Running2 0.13346 16.76 16.247 Running3 16.252 15.621 0.45816

Experiment #1 This method yielded 100% Precision (i.e. all disguised events were classified correctly). Not necessarily representative of the general event detection problem. Future evaluation with more event types, more varied data and a larger set of training and testing data is needed

XML Video Annotation Using the event detection scheme we generate a video description document detailing the event composition of a specific video sequence This XML document annotation may be replaced by a more robust computer-understandable format (e.g. the VEML video event ontology language). <?xml version="1.0" encoding="UTF-8"?> <videoclip> <Filename>H:\Research\MainEvent\ Movies\test_runningandwaving.AVI</Filename> <Length>600</Length> <Event> <Name>unknown</Name> <Start>1</Start> <Duration>106</Duration> </Event> <Name>walking</Name> <Start>107</Start> <Duration>6</Duration> </videoclip>

Video Analysis Tool Takes annotation document as input and organizes the corresponding video segment accordingly. Functions as an aid to a surveillance analyst searching for “Suspicious” events within a stream of video data. Activity of interest may be defined dynamically by the analyst during the running of the utility and flagged for analysis.

Summary and Directions We have proposed an event representation, comparison and detection scheme. Working toward bridging the semantic gap and enabling more efficient video analysis More rigorous experimental testing of concepts Refine event classification through use of multiple machine learning algorithm (e.g. neural networks, decision trees, etc…). Experimentally determine optimal algorithm. Develop a model allowing definition of simultaneous events within the same video sequence Define an access control model that will allow access to surveillance video data to be restricted based on semantic content of video objects Biometrics applications Privacy preserving surveillance

Access Control and Biometrics Control access based on content, association, time etc. Biometrics Restrict access based on semantic content of video rather then low-level features Behavioral type access instead of “fingerprint” Used in combination with other biometric methods

Privacy Preserving Surveillance - Introduction A recent survey at Times Square found 500 visible surveillance cameras in the area and a total of 2500 in New York City. What this essentially means is that, we have scores of surveillance video to be inspected manually by security personnel We need to carry out surveillance but at the same time ensure the privacy of individuals who are good citizens

System Use Raw video surveillance data Face Detection and Face Derecognizing system Suspicious people found Faces of trusted people derecognized to preserve privacy Suspicious events found Comprehensive security report listing suspicious events and people detected Suspicious Event Detection System Manual Inspection of video data Report of security personnel

System Architecture Input Video Finding location of the face in the image Breakdown input video into sequence of images Perform Segmentation Compare face to trusted and untrusted database Raise an alarm that a potential intruder was detected Potential intruder found Trusted face found Derecognize the face in the image

Acknowledgements Prof. Latifur Khan Gal Lavee (Surveillance and access control) Ryan Layfield (Consultant to project) Sai Chaitanya (Privacy) Parveen Pallabi (Biometrics)