1. Data Mining for Surveillance Applications Suspicious Event Detection
Dr. Bhavani Thuraisingham
December 1, 2008

2. Problems Addressed
Huge amounts of video data available in the security domain
Analysis is being done off-line usually using “Human Eyes”
Need for tools to aid human analyst ( pointing out areas in video where unusual activity occurs)
Consider corporate security for a fenced section of sensitive property
The guard suspects there may have been a breach of the perimeter fence at some point during the last 48 hours
They must:
Manually review 48 hours of tape
Consider multiple cameras and camera angles
Distinguish between normal personnel and intruders

3. Example Using our proposed system:
Greatly Increase video analysis efficiency
User Defined
Event of interest
Video Data
Annotated Video w/ events of interest highlighted

4. The Semantic Gap
The disconnect between the low-level features a machine sees when a video is input into it and the high-level semantic concepts (or events) a human being sees when looking at a video clip
Low-Level features: color, texture, shape
High-level semantic concepts: presentation, newscast, boxing match

5. Our Approach Event Representation Event Comparison Event Detection
Estimate distribution of pixel intensity change
Event Comparison
Contrast the event representation of different video sequences to determine if they contain similar semantic event content.
Event Detection
Using manually labeled training video sequences to classify unlabeled video sequences

6. Event Representation
Measures the quantity and type of changes occurring within a scene
A video event is represented as a set of x, y and t intensity gradient histograms over several temporal scales.
Histograms are normalized and smoothed

7. Event Comparison
Determine if the two video sequences contain similar high-level semantic concepts (events).
Produces a number that indicates how close the two compared events are to one another.
The lower this number is the closer the two events are.

8. Event Detection A robust event detection system should be able to
Recognize an event with reduced sensitivity to actor (e.g. clothing or skin tone) or background lighting variation.
Segment an unlabeled video containing multiple events into event specific segments

9. Walking1 Running1 Waving2
Labeled Video Events
These events are manually labeled and used to classify unknown events
Walking1 Running1 Waving2

10. Labeled Video Events walking1 walking2 walking3 running1 running2
walking1
walking2
walking3
running1
running2
running3
running4
waving 2
1.2262
1.383
1.3791
10.961
1.4757
1.5003
1.2908
1.541
10.581
1.1298
1.0933
1.1221
10.231
14.469
15.05
14.2
15.607
waving2

11. Experiment #1
Problem: Recognize and classify events irrespective of direction (right-to-left, left-to-right) and with reduced sensitivity to spatial variations (Clothing)
“Disguised Events”- Events similar to testing data except subject is dressed differently
Compare Classification to “Truth” (Manual Labeling)

12. Classification: Walking
Experiment #1
Disguised Walking 1
walking1
walking2
walking3
running1
running2
running3
running4
waving2
1.5476
1.4633
1.5724
1.5406
12.225
Classification: Walking

13. Classification: Walking
Experiment #1
Disguised Walking 2
walking1
walking2
walking3
running1
running2
running3
running4
waving2
0.948
1.9412
1.844
1.8711
1.9673
10.191
Classification: Walking

14. Classification: Running
Experiment #1
Disguised Running 1
walking1
walking2
walking3
running1
running2
running3
running4
waving2
1.411
1.3841
1.0637
1.0957
11.629
Classification: Running

15. Classification: Running
Experiment #1
Disguised Running 2
walking1
walking2
walking3
running1
running2
running3
running4
waving2
1.3543
1.1909
1.0071
13.141
Classification: Running

16. Classifying Disguised Events
Disguised Running 3
walking1
walking2
walking3
running1
running2
running3
running4
waving2
1.3049
1.0021
0.8114
1.1042
1.1189
1.0902
12.801
Classification: Running

17. Classifying Disguised Events
Disguised Waving 1
walking1
walking2
walking3
running1
running2
running3
running4
waving2
13.646
13.113
13.452
18.615
19.592
18.621
20.239
2.2451
Classification: Waving

18. Classifying Disguised Events
Disguised Waving 2
walking1
walking2
walking3
running1
running2
running3
running4
waving2
12.792
12.132
12.681
18.104
18.956
18.029
19.547
3.1336
Classification: Waving

19. Classifying Disguised Events
Disguise
walking1
walking2
running1
running2
running3
waving1
waving2
1.2159
14.471
13.429
1.4317
1.1824
12.295
11.29
15.266
15.007
Running2
16.76
16.247
Running3
16.252
15.621

20. Experiment #1
This method yielded 100% Precision (i.e. all disguised events were classified correctly).
Not necessarily representative of the general event detection problem.
Future evaluation with more event types, more varied data and a larger set of training and testing data is needed

21. XML Video Annotation
Using the event detection scheme we generate a video description document detailing the event composition of a specific video sequence
This XML document annotation may be replaced by a more robust computer-understandable format (e.g. the VEML video event ontology language).
<?xml version="1.0" encoding="UTF-8"?>
<videoclip>
<Filename>H:\Research\MainEvent\
Movies\test_runningandwaving.AVI</Filename>
<Length>600</Length>
<Event>
<Name>unknown</Name>
<Start>1</Start>
<Duration>106</Duration>
</Event>
<Name>walking</Name>
<Start>107</Start>
<Duration>6</Duration>
</videoclip>

22. Video Analysis Tool
Takes annotation document as input and organizes the corresponding video segment accordingly.
Functions as an aid to a surveillance analyst searching for “Suspicious” events within a stream of video data.
Activity of interest may be defined dynamically by the analyst during the running of the utility and flagged for analysis.

23. Summary and Directions
We have proposed an event representation, comparison and detection scheme.
Working toward bridging the semantic gap and enabling more efficient video analysis
More rigorous experimental testing of concepts
Refine event classification through use of multiple machine learning algorithm (e.g. neural networks, decision trees, etc…). Experimentally determine optimal algorithm.
Develop a model allowing definition of simultaneous events within the same video sequence
Define an access control model that will allow access to surveillance video data to be restricted based on semantic content of video objects
Biometrics applications
Privacy preserving surveillance

24. Access Control and Biometrics
Control access based on content, association, time etc.
Biometrics
Restrict access based on semantic content of video rather then low-level features
Behavioral type access instead of “fingerprint”
Used in combination with other biometric methods

25. Privacy Preserving Surveillance - Introduction
A recent survey at Times Square found 500 visible surveillance cameras in the area and a total of 2500 in New York City.
What this essentially means is that, we have scores of surveillance video to be inspected manually by security personnel
We need to carry out surveillance but at the same time ensure the privacy of individuals who are good citizens

26. System Use Raw video surveillance data
Face Detection and Face Derecognizing system
Suspicious people found
Faces of trusted people derecognized to preserve privacy
Suspicious events found
Comprehensive security report listing suspicious events and people detected
Suspicious Event Detection System
Manual Inspection of video data
Report of security personnel

27. System Architecture Input Video
Finding location of the face in the image
Breakdown input video into sequence of images
Perform Segmentation
Compare face to trusted and untrusted database
Raise an alarm that a potential intruder was detected
Potential intruder found
Trusted face found
Derecognize the face in the image

28. Acknowledgements Prof. Latifur Khan
Gal Lavee (Surveillance and access control)
Ryan Layfield (Consultant to project)
Sai Chaitanya (Privacy)
Parveen Pallabi (Biometrics)