1. Data Mining for Surveillance Applications Suspicious Event Detection Dr. Bhavani Thuraisingham

2. Problems Addressed Huge amounts of video data available in the security domain Huge amounts of video data available in the security domain Analysis is being done off-line usually using “Human Eyes” Analysis is being done off-line usually using “Human Eyes” Need for tools to aid human analyst ( pointing out areas in video where unusual activity occurs) Need for tools to aid human analyst ( pointing out areas in video where unusual activity occurs) Consider corporate security for a fenced section of sensitive property Consider corporate security for a fenced section of sensitive property The guard suspects there may have been a breach of the perimeter fence at some point during the last 48 hours The guard suspects there may have been a breach of the perimeter fence at some point during the last 48 hours They must: They must: Manually review 48 hours of tape Manually review 48 hours of tape Consider multiple cameras and camera angles Consider multiple cameras and camera angles Distinguish between normal personnel and intruders Distinguish between normal personnel and intruders

3. Example Using our proposed system: Using our proposed system: Greatly Increase video analysis efficiency Greatly Increase video analysis efficiency User Defined Event of interest Video Data Annotated Video w/ events of interest highlighted

4. The Semantic Gap The disconnect between the low-level features a machine sees when a video is input into it and the high- level semantic concepts (or events) a human being sees when looking at a video clip The disconnect between the low-level features a machine sees when a video is input into it and the high- level semantic concepts (or events) a human being sees when looking at a video clip Low-Level features: color, texture, shape Low-Level features: color, texture, shape High-level semantic concepts: presentation, newscast, boxing match High-level semantic concepts: presentation, newscast, boxing match

5. Our Approach Event Representation Event Representation Estimate distribution of pixel intensity change Estimate distribution of pixel intensity change Event Comparison Event Comparison Contrast the event representation of different video sequences to determine if they contain similar semantic event content. Contrast the event representation of different video sequences to determine if they contain similar semantic event content. Event Detection Event Detection Using manually labeled training video sequences to classify unlabeled video sequences Using manually labeled training video sequences to classify unlabeled video sequences

6. Event Representation Measures the quantity and type of changes occurring within a scene Measures the quantity and type of changes occurring within a scene A video event is represented as a set of x, y and t intensity gradient histograms over several temporal scales. A video event is represented as a set of x, y and t intensity gradient histograms over several temporal scales. Histograms are normalized and smoothed Histograms are normalized and smoothed

7. Event Comparison Determine if the two video sequences contain similar high-level semantic concepts (events). Determine if the two video sequences contain similar high-level semantic concepts (events). Produces a number that indicates how close the two compared events are to one another. Produces a number that indicates how close the two compared events are to one another. The lower this number is the closer the two events are. The lower this number is the closer the two events are.

8. Event Detection A robust event detection system should be able to A robust event detection system should be able to Recognize an event with reduced sensitivity to actor (e.g. clothing or skin tone) or background lighting variation. Recognize an event with reduced sensitivity to actor (e.g. clothing or skin tone) or background lighting variation. Segment an unlabeled video containing multiple events into event specific segments Segment an unlabeled video containing multiple events into event specific segments

9. Labeled Video Events These events are manually labeled and used to classify unknown events These events are manually labeled and used to classify unknown events Walking1 Running1Waving2

10. Labeled Video Events walking1walking2walking3running1running2running3running4 waving 2 walking100.276250.245081.22621.3830.974721.379110.961 walking20.2762500.178881.47571.50031.29081.54110.581 walking30.245080.1788801.12981.09330.886041.122110.231 running11.22621.47571.129800.438290.304510.3982314.469 running21.3831.50031.09330.4382900.238040.1076115.05 running30.974721.29080.886040.304510.2380400.2048914.2 running41.37911.5411.12210.398230.107610.20489015.607 waving210.96110.58110.23114.46915.0514.215.6070

11. Experiment #1 Problem: Recognize and classify events irrespective of direction (right-to-left, left-to-right) and with reduced sensitivity to spatial variations (Clothing) Problem: Recognize and classify events irrespective of direction (right-to-left, left-to-right) and with reduced sensitivity to spatial variations (Clothing) “Disguised Events”- Events similar to testing data except subject is dressed differently “Disguised Events”- Events similar to testing data except subject is dressed differently Compare Classification to “Truth” (Manual Labeling) Compare Classification to “Truth” (Manual Labeling)

12. Experiment #1 Classification: Walking Disguised Walking 1walking1walking2walking3running1running2running3running4waving20.976530.451540.596081.54761.46331.57241.540612.225

13. Experiment #1 Classification: Walking Disguised Walking 2walking1walking2walking3running1running2running3running4waving20.9480.380970.538521.94121.8441.87111.967310.191

14. Experiment #1 Classification: Running Disguised Running 1walking1walking2walking3running1running2running3running4waving21.4111.38411.06370.567240.974170.935871.095711.629

15. Experiment #1 Classification: Running Disguised Running 2walking1walking2walking3running1running2running3running4waving21.35431.19091.00710.615410.958330.942270.9373113.141

16. Classifying Disguised Events Classification: Running Disguised Running 3walking1walking2walking3running1running2running3running4waving21.30491.00210.880920.81141.10421.11891.090212.801

17. Classifying Disguised Events Classification: Waving Disguised Waving 1walking1walking2walking3running1running2running3running4waving213.64613.11313.45218.61519.59218.62120.2392.2451

18. Classifying Disguised Events Classification: Waving Disguised Waving 2walking1walking2walking3running1running2running3running4waving212.79212.13212.68118.10418.95618.02919.5473.1336

19. Classifying Disguised Events Disguisewalking1Disguisewalking2Disguiserunning1Disguiserunning2Disguiserunning3Disguisewaving1Disguisewaving2 Disguisewalking100.193391.21590.859380.6757714.47113.429 Disguisewalking20.1933901.43171.18240.9558212.29511.29 Disguiserunning11.21591.431700.375920.4518715.26615.007 DisguiseRunning20.859381.18240.3759200.1334616.7616.247 DisguiseRunning30.675770.955820.451870.13346016.25215.621 Disguisewaving114.47112.29515.26616.7616.25200.45816 Disguisewaving213.42911.2915.00716.24715.6210.458160

20. Experiment #1 This method yielded 100% Precision (i.e. all disguised events were classified correctly). This method yielded 100% Precision (i.e. all disguised events were classified correctly). Not necessarily representative of the general event detection problem. Not necessarily representative of the general event detection problem. Future evaluation with more event types, more varied data and a larger set of training and testing data is needed Future evaluation with more event types, more varied data and a larger set of training and testing data is needed

21. XML Video Annotation Using the event detection scheme we generate a video description document detailing the event composition of a specific video sequence Using the event detection scheme we generate a video description document detailing the event composition of a specific video sequence This XML document annotation may be replaced by a more robust computer-understandable format (e.g. the VEML video event ontology language). This XML document annotation may be replaced by a more robust computer-understandable format (e.g. the VEML video event ontology language). <videoclip> H:\Research\MainEvent\ H:\Research\MainEvent\ Movies\test_runningandwaving.AVI Movies\test_runningandwaving.AVI 600 600 unknown unknown 1 1 106 106 walking walking 107 107 6 6 </videoclip>

22. Video Analysis Tool Takes annotation document as input and organizes the corresponding video segment accordingly. Takes annotation document as input and organizes the corresponding video segment accordingly. Functions as an aid to a surveillance analyst searching for “Suspicious” events within a stream of video data. Functions as an aid to a surveillance analyst searching for “Suspicious” events within a stream of video data. Activity of interest may be defined dynamically by the analyst during the running of the utility and flagged for analysis. Activity of interest may be defined dynamically by the analyst during the running of the utility and flagged for analysis.

23. Summary and Directions We have proposed an event representation, comparison and detection scheme. We have proposed an event representation, comparison and detection scheme. Working toward bridging the semantic gap and enabling more efficient video analysis Working toward bridging the semantic gap and enabling more efficient video analysis More rigorous experimental testing of concepts More rigorous experimental testing of concepts Refine event classification through use of multiple machine learning algorithm (e.g. neural networks, decision trees, etc…). Experimentally determine optimal algorithm. Refine event classification through use of multiple machine learning algorithm (e.g. neural networks, decision trees, etc…). Experimentally determine optimal algorithm. Develop a model allowing definition of simultaneous events within the same video sequence Develop a model allowing definition of simultaneous events within the same video sequence Define an access control model that will allow access to surveillance video data to be restricted based on semantic content of video objects Define an access control model that will allow access to surveillance video data to be restricted based on semantic content of video objects Biometrics applications Biometrics applications Privacy preserving surveillance Privacy preserving surveillance

24. Access Control and Biometrics Access Control Access Control Control access based on content, association, time etc. Control access based on content, association, time etc. Biometrics Biometrics Restrict access based on semantic content of video rather then low-level features Restrict access based on semantic content of video rather then low-level features Behavioral type access instead of “fingerprint” Behavioral type access instead of “fingerprint” Used in combination with other biometric methods Used in combination with other biometric methods

25. Privacy Preserving Surveillance - Introduction A recent survey at Times Square found 500 visible surveillance cameras in the area and a total of 2500 in New York City. What this essentially means is that, we have scores of surveillance video to be inspected manually by security personnel We need to carry out surveillance but at the same time ensure the privacy of individuals who are good citizens

26. System Use Raw video surveillance data Face Detection and Face Derecognizing system Suspicious Event Detection System Manual Inspection of video data Comprehensive security report listing suspicious events and people detected Suspicious people found Suspicious events found Report of security personnel Faces of trusted people derecognized to preserve privacy

27. System Architecture Input Video Breakdown input video into sequence of images Perform Segmentation Compare face to trusted and untrusted database Finding location of the face in the image Derecognize the face in the image Raise an alarm that a potential intruder was detected Trusted face found Potential intruder found

28. Acknowledgements Prof. Latifur Khan Prof. Latifur Khan Gal Lavee (Surveillance and access control) Gal Lavee (Surveillance and access control) Ryan Layfield (Consultant to project) Ryan Layfield (Consultant to project) Sai Chaitanya (Privacy) Sai Chaitanya (Privacy) Parveen Pallabi (Biometrics) Parveen Pallabi (Biometrics)