Misuse and Anomaly Detection Sampath Kannan Wenke Lee Insup Lee Diana Spears Oleg Sokolsky William Spears Linda Zhao

Network Intrusion Detection Systems (NIDS) oImportant defense to protect sensitive information and resources on the network. oUsually have the following functionalities. o Observe traffic and extract features o Pattern match with database of “attack signatures” to detect misuse (intrusion) oObserve statistical properties and check against specifications of correct behavior to detect anomalies

Shortcomings of Current NIDS oNew attack strategies arise constantly and attack signature databases become obsolete rapidly. oVolume and interleaving of traffic at backbone of network makes complex signature recognition infeasible.

Shortcomings cont’d o Anomaly detection algorithms are primitive. We want more scalable yet more sophisticated techniques. o Want to reduce the number of false positives in anomaly detection to make it useful.

Our Approach oUse Machine Learning, Data Mining, and Case-Based Reasoning techniques to learn new intruder models on the fly. oBuild a taxonomy of possible anomalies; extract relevant features; use statistical and machine learning techniques to reduce false-alarm rate.

Our Approach – Cont’d oApply sophisticated algorithms designed in the resource-constrained data stream model to NIDS. oIntegrate all of these modules into a MaC- based system architecture.

Existing Infrastructure oMonitoring and Checking (MaC) architecture for run-time monitoring oUser specified instrumentation of running programs to extract important state changes. (Primitive Event Definition Language (PEDL)). oUser specified conversion of these low-level events to abstract events relevant to properties (MEDL). oChecker for processing abstract event stream to monitor correctness.

Existing Infrastructure – Cont’d oAn experimental test-bed to test performance of Intrusion and Anomaly Detection Systems. o Enhancement of a similar set-up from MIT Lincoln Labs from the 90’s. o Models hacker profiles and taxonomy of attacks and generates “realistic” normal and attack traffic. o Metrics for evaluating potency of attacks.

Using MaC for NIDS oNeed multiple Primitive Event Definition Languages (PEDLs) to model different algorithmic techniques for extracting abstract events. oNeed dynamically changeable properties as machine learning approaches discover new attack signatures. oNeed integration module that combines the results of various modules.

Inferring Mixtures of Markov Chains Batu, Guha, Kannan A theoretical result...

An example oNetwork traffic log … each party behaves like a Markov Chain oSome parties are malicious oCan you tease out the malicious chains from a single common log?

Another Example: Browsing habits oYou read sports and cartoons. You’re equally likely to read both. You do not remember what you read last. oYou’d expect a “random” sequence SCSSCSSCSSCCSCCCSSSSCSC…

Suppose there are two oI like health, entertainment, and fashion oI always read entertainment first, health next and fashion last oThe sequence would be EHFEHFEHFEHFEHFEHFEHF…

Two readers, one log file oIf there is one log file… oAssume there is no correlation between us SECHSSFECSHFESCSSHCFCESCHCCFSESHFESSHFE… Is there enough information to tell that there are two people browsing? What are they browsing? How are they browsing?

Clues in stream? oYes! (under model assumptions). oH, E, F have special relationship. oThey cannot belong to different (uncorrelated) people. oNot clear about S and C... Could be 3 uncorrelated persons. SECHSSFECSHFESCSSHCFCESCHCCFSESHFESSHFE…

Markov Chains as Stochastic Sources Output sequence:

Markov chains on S,E,C,H,F S C 1/2 Modeled by … H 1 E F 1 1

Problem Statement (informal) oTwo or more probabilistic processes oWe are observing interleaved behavior oWe do not know which state belongs to which process – cold start.

The Problem MC1 MC Observe Infer: MC1, MC2, & mixing parameters

For our problem we assume: Stream is polynomially long in the number of states of each Markov chain (need perhaps long stream). C : maximum cover time Q : upper bound on the denominator of any probability Nonzero probabilities are bounded away from 0. Space available is some small polynomial in #states. Under these assumptions, we can identify individual chains if their state spaces are disjoint.

Research Directions oMany exciting directions oOur research team has expertise in network security, machine learning, AI, real-time systems, and algorithm design oWe expect interesting synergies between these strengths.