1. 1 Multimedia Systems Security: Video Data Analysis for Security Applications and Securing Video Data Dr. Bhavani Thuraisingham September 2007

2. 2 Outline Data Mining for Security Applications Data Mining for Security Applications Video Analysis Suspicious Event Detection Video Analysis Suspicious Event Detection Access Control Access Control Privacy Preserving Surveillance Privacy Preserving Surveillance Secure Third Party Publication of Video Data Secure Third Party Publication of Video Data Malicious Code Detection Malicious Code Detection Directions and Opportunities Directions and Opportunities

3. 3 Acknowledgments Professor Latifur Khan for data mining applications and Malicious Code Detection Professor Latifur Khan for data mining applications and Malicious Code Detection Prof Elisa Bertino (Purdue) and Prof. Jianping Fan (UNCC) for Privacy Preserving Video Analysis Prof Elisa Bertino (Purdue) and Prof. Jianping Fan (UNCC) for Privacy Preserving Video Analysis Prof. Elisa Bertino, Prof Elena Ferrari (Milan/Como) and Prof. Barbara Carminati (Milan/Como) for Secure Third Party Publication Prof. Elisa Bertino, Prof Elena Ferrari (Milan/Como) and Prof. Barbara Carminati (Milan/Como) for Secure Third Party Publication Students at the University of Texas at Dallas Students at the University of Texas at Dallas

4. 4 Data Mining for Security Applications Data Mining has many applications in Cyber Security and National Security Data Mining has many applications in Cyber Security and National Security Intrusion detection, worm detection, firewall policy management Intrusion detection, worm detection, firewall policy management Counter-terrorism applications and Surveillance Counter-terrorism applications and Surveillance Fraud detection, Insider threat analysis Fraud detection, Insider threat analysis Need to enforce security but at the same time ensure privacy Need to enforce security but at the same time ensure privacy

5. 5 Problems Addressed Huge amounts of video data available in the security domain Huge amounts of video data available in the security domain Analysis is being done off-line usually using “Human Eyes” Analysis is being done off-line usually using “Human Eyes” Need for tools to aid human analyst ( pointing out areas in video where unusual activity occurs) Need for tools to aid human analyst ( pointing out areas in video where unusual activity occurs) Need to control access to the video data Need to control access to the video data Need to securely publish video data Need to securely publish video data Need to ensure that the data is not maliciously corrpupted Need to ensure that the data is not maliciously corrpupted

6. 6 Video Analysis fore Security The Semantic Gap The disconnect between the low-level features a machine sees when a video is input into it and the high- level semantic concepts (or events) a human being sees when looking at a video clip The disconnect between the low-level features a machine sees when a video is input into it and the high- level semantic concepts (or events) a human being sees when looking at a video clip Low-Level features: color, texture, shape Low-Level features: color, texture, shape High-level semantic concepts: presentation, newscast, boxing match High-level semantic concepts: presentation, newscast, boxing match

7. 7 Our Approach Event Representation Event Representation Estimate distribution of pixel intensity change Estimate distribution of pixel intensity change Event Comparison Event Comparison Contrast the event representation of different video sequences to determine if they contain similar semantic event content. Contrast the event representation of different video sequences to determine if they contain similar semantic event content. Event Detection Event Detection Using manually labeled training video sequences to classify unlabeled video sequences Using manually labeled training video sequences to classify unlabeled video sequences

8. 8 Event Representation, Comparison, Detection Measures the quantity and type of changes occurring within a scene Measures the quantity and type of changes occurring within a scene A video event is represented as a set of x, y and t intensity gradient histograms over several temporal scales. A video event is represented as a set of x, y and t intensity gradient histograms over several temporal scales. Histograms are normalized and smoothed Histograms are normalized and smoothed Determine if the two video sequences contain similar high-level semantic concepts (events). Determine if the two video sequences contain similar high-level semantic concepts (events). Produces a number that indicates how close the two compared events are to one another. Produces a number that indicates how close the two compared events are to one another. The lower this number is the closer the two events are. The lower this number is the closer the two events are. A robust event detection system should be able to A robust event detection system should be able to Recognize an event with reduced sensitivity to actor (e.g. clothing or skin tone) or background lighting variation. Recognize an event with reduced sensitivity to actor (e.g. clothing or skin tone) or background lighting variation. Segment an unlabeled video containing multiple events into event specific segments Segment an unlabeled video containing multiple events into event specific segments

9. 9 Labeled Video Events These events are manually labeled and used to classify unknown events These events are manually labeled and used to classify unknown events Walking1 Running1Waving2

10. 10 Labeled Video Events walking1walking2walking3running1running2running3running4 waving 2 walking100.276250.245081.22621.3830.974721.379110.961 walking20.2762500.178881.47571.50031.29081.54110.581 walking30.245080.1788801.12981.09330.886041.122110.231 running11.22621.47571.129800.438290.304510.3982314.469 running21.3831.50031.09330.4382900.238040.1076115.05 running30.974721.29080.886040.304510.2380400.2048914.2 running41.37911.5411.12210.398230.107610.20489015.607 waving210.96110.58110.23114.46915.0514.215.6070

11. 11 Experiment #1 Problem: Recognize and classify events irrespective of direction (right-to-left, left-to-right) and with reduced sensitivity to spatial variations (Clothing) Problem: Recognize and classify events irrespective of direction (right-to-left, left-to-right) and with reduced sensitivity to spatial variations (Clothing) “Disguised Events”- Events similar to testing data except subject is dressed differently “Disguised Events”- Events similar to testing data except subject is dressed differently Compare Classification to “Truth” (Manual Labeling) Compare Classification to “Truth” (Manual Labeling)

12. 12 Experiment #1 Classification: Walking Disguised Walking 1walking1walking2walking3running1running2running3running4waving20.976530.451540.596081.54761.46331.57241.540612.225

13. 13 Experiment #1 Classification: Running Disguised Running 1walking1walking2walking3running1running2running3running4waving21.4111.38411.06370.567240.974170.935871.095711.629

14. 14 XML Video Annotation Using the event detection scheme we generate a video description document detailing the event composition of a specific video sequence Using the event detection scheme we generate a video description document detailing the event composition of a specific video sequence This XML document annotation may be replaced by a more robust computer-understandable format (e.g. the VEML video event ontology language). This XML document annotation may be replaced by a more robust computer-understandable format (e.g. the VEML video event ontology language). <videoclip> H:\Research\MainEvent\ H:\Research\MainEvent\ Movies\test_runningandwaving.AVI Movies\test_runningandwaving.AVI 600 600 unknown unknown 1 1 106 106 walking walking 107 107 6 6 </videoclip>

15. 15 Video Analysis Tool Takes annotation document as input and organizes the corresponding video segment accordingly. Takes annotation document as input and organizes the corresponding video segment accordingly. Functions as an aid to a surveillance analyst searching for “Suspicious” events within a stream of video data. Functions as an aid to a surveillance analyst searching for “Suspicious” events within a stream of video data. Activity of interest may be defined dynamically by the analyst during the running of the utility and flagged for analysis. Activity of interest may be defined dynamically by the analyst during the running of the utility and flagged for analysis.

16. 16 Access Control: Authorization Objects Authorization objects, the actual video data to which we wish to restrict access and represented in the form of a 7 value tuple. Authorization objects, the actual video data to which we wish to restrict access and represented in the form of a 7 value tuple. This tuple contains information about the content of a particular video object. Some of this content information pertains to high-level semantic information such as events and objects. This tuple contains information about the content of a particular video object. Some of this content information pertains to high-level semantic information such as events and objects. This information is stored as a set of concepts taken from a “closed-world” hierarchical taxonomy which relates these concepts to one another. This information is stored as a set of concepts taken from a “closed-world” hierarchical taxonomy which relates these concepts to one another. Other content information such as location and timestamp is represented as a special data type that allows more meaningful specification of this unique kind of content. Other content information such as location and timestamp is represented as a special data type that allows more meaningful specification of this unique kind of content.

17. 17 Access Control: Video Object Hierarchy Surveillance Object Still Camera Video Camera Satellite Image Aerial ImageHallway Camera Lobby Camera

18. 18 Access Control: Other Concepts Events is the set of semantic events occurring within the video object. Events is the set of semantic events occurring within the video object. Objects is the set of semantic objects contained within the video object. Objects is the set of semantic objects contained within the video object. Location is the term indicating the geographic earth coordinates of where the surveillance video object was captured. Location is the term indicating the geographic earth coordinates of where the surveillance video object was captured. Timestamp is the term describing the real world time when the video was captured. Timestamp is the term describing the real world time when the video was captured.

19. 19 Access Control: Event and Object Hhierarchies Video Event Stationar y Event Mobile Event Waving Walkin g Runni ng Jumping Video Object Vehicle Toy Truck BallFrisbe e Car

20. 20 Video Object Expressions Video object expressions describe the object for which access control is to be applied. Video object expressions describe the object for which access control is to be applied. These expressions are expanded and made more robust so that a video object may be specified not only by its object ID but rather by any of its attributes or their combination. These expressions are expanded and made more robust so that a video object may be specified not only by its object ID but rather by any of its attributes or their combination. This is similar to querying a relational database using a complex SQL query specifying a particular set of records. This is similar to querying a relational database using a complex SQL query specifying a particular set of records. We use access functions to reference the different components of our surveillance video objects for use in our expressions. We use access functions to reference the different components of our surveillance video objects for use in our expressions.

21. 21 Authorization Subjects We use the concept of user credentials to authorize users. We use the concept of user credentials to authorize users. That is, each user entity, in addition to having a unique user id or belonging to a group also possesses a set of credentials. That is, each user entity, in addition to having a unique user id or belonging to a group also possesses a set of credentials. Each credential is an instantiation of a certain credential type, the template for credentials in which the set of credential attributes, and whether they are optional or obligatory is defined. Each credential is an instantiation of a certain credential type, the template for credentials in which the set of credential attributes, and whether they are optional or obligatory is defined. Specific values are assigned to these attributes when a new user instantiates the credential type. Specific values are assigned to these attributes when a new user instantiates the credential type. A subject may instantiate any number of credential types. A subject may instantiate any number of credential types. These credential types are defined in a credential type hierarchy relating each credential type to the other credential types These credential types are defined in a credential type hierarchy relating each credential type to the other credential types

22. 22 Access Control: Credential Type Hierarchy Person Maintenance Staff Security Officer Database Administrator PoliceGuard PatrolmanCaptain

23. 23 Access Control: Authorizations Authorizations are what allow us to specify our access control policy for the objects in our video surveillance database. Authorizations are what allow us to specify our access control policy for the objects in our video surveillance database. Derived Authorizations: The properties of the hierarchical taxonomies used in defining surveillance video object types, semantic event types and semantic object types can be used to obtain implicit authorizations from the explicit authorizations specified as a part of the access control policy base. Derived Authorizations: The properties of the hierarchical taxonomies used in defining surveillance video object types, semantic event types and semantic object types can be used to obtain implicit authorizations from the explicit authorizations specified as a part of the access control policy base. Additionally the relationships between the various privilege modes allow further extrapolation of authorizations. Additionally the relationships between the various privilege modes allow further extrapolation of authorizations.

24. 24 Access Control Algorithm User requests for surveillance video objects must be compared to the policy base of object authorizations before access can be granted. User requests for surveillance video objects must be compared to the policy base of object authorizations before access can be granted. Furthermore, if the user request is not for a specific object but rather a query for a particular set of objects the system must be able to successfully reconcile the query criteria with the objects existing in the database. Furthermore, if the user request is not for a specific object but rather a query for a particular set of objects the system must be able to successfully reconcile the query criteria with the objects existing in the database. If the user request is authorized for some part (but not all) of the surveillance video object instead of denying the access entirely it is possible to post-process the data after retrieval and release only authorized portions to the user. If the user request is authorized for some part (but not all) of the surveillance video object instead of denying the access entirely it is possible to post-process the data after retrieval and release only authorized portions to the user. Hence our access control process has three major components: Authorization, retrieval, post-processing and delivery. Hence our access control process has three major components: Authorization, retrieval, post-processing and delivery.

25. 25 Access Control Policies: Extensions Policies based on content, associations, time, and event Policies based on content, associations, time, and event Policy engine that evaluates the policies for consistency Policy engine that evaluates the policies for consistency Enforcement engine for enforcing the policies Enforcement engine for enforcing the policies Distributed policies: Objects at different locations taken together are sensitive Distributed policies: Objects at different locations taken together are sensitive

26. 26 System Architecture for Access Control User Pull/Query Push/result Video XML Documents X-AccessX-Admin Admin Tools Policy base Credential base

27. 27 Third-Party Architecture Credential base policy base XML Source User/Subject Owner Publisher Query Reply document SE-XML credentials The Owner is the producer of information It specifies access control policies on the Video objects The Owner is the producer of information It specifies access control policies on the Video objects The Publisher is responsible for managing (a portion of) the Owner information and answering subject queries The Publisher is responsible for managing (a portion of) the Owner information and answering subject queries Goal: Untrusted Publisher with respect to Authenticity and Completeness checking Goal: Untrusted Publisher with respect to Authenticity and Completeness checking

28. 28 Policy Information Merkle Signature XML Document SE-XML Document Security Enhanced Video XML document

29. Privacy Preserving Video Analysis A recent survey at Times Square found 500 visible surveillance cameras in the area and a total of 2500 in New York City. What this essentially means is that, we have scores of surveillance video to be inspected manually by security personnel We need to carry out surveillance but at the same time ensure the privacy of individuals who are good citizens

30. 30 System Use Raw video surveillance data Face Detection and Face Derecognizing system Suspicious Event Detection System Manual Inspection of video data Comprehensive security report listing suspicious events and people detected Suspicious people found Suspicious events found Report of security personnel Faces of trusted people derecognized to preserve privacy

31. 31 Detecting Malicious Code ✗ Content -based approaches consider only machine-codes (byte-codes). ✗ Is it possible to consider higher-level source codes for malicious code detection? ✗ Yes: Diassemble the binary executable and retrieve the assembly program ✗ Extract important features from the assembly program ✗ Combine with machine-code features ✗ Extract both Binary n-gram features and Assembly n-gram features

32. 32 Hybrid Feature Retrieval (HFR) Training Training Testing

33. 33 Summary and Directions We have proposed an event representation, comparison and detection scheme. We have proposed an event representation, comparison and detection scheme. Working toward bridging the semantic gap and enabling more efficient video analysis Working toward bridging the semantic gap and enabling more efficient video analysis More rigorous experimental testing of concepts More rigorous experimental testing of concepts Refine event classification through use of multiple machine learning algorithm (e.g. neural networks, decision trees, etc…). Experimentally determine optimal algorithm. Refine event classification through use of multiple machine learning algorithm (e.g. neural networks, decision trees, etc…). Experimentally determine optimal algorithm. Develop a model allowing definition of simultaneous events within the same video sequence Develop a model allowing definition of simultaneous events within the same video sequence Define an access control model that will allow access to surveillance video data to be restricted based on semantic content of video objects Define an access control model that will allow access to surveillance video data to be restricted based on semantic content of video objects Secure publishing of Video Documents Secure publishing of Video Documents Privacy Preserving Analysis Privacy Preserving Analysis Detecting Malicious Code Detecting Malicious Code

34. 34 Opportunities for the Community We We