Formal Methods for Intrusion Detection Presented by Brian Kellogg CSE 914: Formal Methods for Software Development Michigan State University December 11 th, 2002

Purpose and Method Find intrusion detection methods that utilize formal methods Analyze strengths and weaknesses of each method Compare the methods and see if they can be combined in such a way to improve one another Found three research papers on intrusion detection that used formal methods for different purposes

Intrusion Detection Quickie The SANS institute defines intrusion detection as “the art of detecting inappropriate, incorrect, or anomalous activity” Two types: Host-based: detects intrusions on a specific host Network-based: detects intrusions on a network Two (main) methods: Knowledge-based Determine vulnerabilities and attempts to detect vulnerabilities Low false alarm rate Attacks not specified are not detected Behavior-based Determines normal system activity High false alarm rate Able to detect many intrusions (even ones not previously known)

Intrusion Detection Continued Why use intrusion detection, why not just prevent the attacks? Firewalls can prevent many attacks, but have no power over the internal network Certain network activities that have legitimate uses can also signify an attack (e.g. port scans) What should an intrusion system do when it detects an attack? Responses range from s to reconfiguring the network Just because the system detects an intrusion, may be legitimate Severe (or even simple) responses can be utilized by attackers to create new attacks

Yasinsac Paper (Motivation) “An Environment for Security Protocol Intrusion Detection” Traditional methods of protocol analysis not fool proof or complete Different protocols running concurrently can create new exploits Shift to “tunneling” paradigm in networks Sensitive data sent over same links as non- sensitive data Cryptographic techniques must be applied at higher layer (application layer)

Yasinsac Paper (Method) Take knowledge gained from formal analysis of security protocols and make them in to intrusion signatures Uses both knowledge-based and behavior- based intrusion detection Knowledge-based: signature an ordering of activity traces Behavior-based: surveys taxonomies and protocol principles to determine profile strategies and behavior recognition State-based attack recognition

Yasinsac Paper (Method) IKE protocol: A  B: HDR 1, SA A, KE A, N A, A B  A: HDR 2, SA B, KE B, N B, B, {prf(K AB, (KE B, KE A, KE B, KE A, B))}K B Exploit: A  B: HDR 1, SA A, KE A, N A, A I  B: HDR 1, SA A, KE A, N A, I B  I: HDR 2, SA B, KE B, N B, B, {prf(K AB, (KE B, KE A, KE B, KE A, B))}K B

Yasinsac Paper (Architecture) Central monitor, each principal communicates with monitor through secure channel

Pouzol Paper Motivation: Algorithm that detects attacks in a declarative IDS is a black box Partial instances of attacks can choke an IDS Wants to give more power to security officer to choose which attack instances are important Method: Formally specify intrusion signatures and detection rules Create a lattice used to define equivalence classes that defines a signature Choose an equivalence relation that can reduce the number of instances reported

Pouzol Lattice Т {U 1, U 2, T 1, T 2, T 3 } {U 1, U 2, T 3 } {U 1, U 2 } {U 2, T 3 } {T3}{T3} {U 1 }{U 2 } { } U 1 U 2 T 3 : In this equivalence class, every instance that has a unique pair of users and a third time stamp will be reported. This is an example of a good choice. This class will resist the choking attack, and will report all completed instances of an attack. Having the final timestamp means that the last part of the attack occurred, thus only a completed attack is being reported.

NetSTAT Paper (Motivation) “NetSTAT: A Network-based Intrusion Detection Approach” Motivated by the increase of network reliance and attacks Host-based intrusion detection fails to detect these attacks Firewalls do an excellent job of preventing external intrusions, but internal threats are left unchecked

NetSTAT Paper (Method) NetSTAT is a network-based intrusion detection system Wants to solve: Networks generate large amounts of data Some attacks occur only in a certain portion of a network Too much communication between IDS components can clog a network Networks can grow very large Able to work with host-based methods Four components: A network fact base A state transition scenario database Many general purpose probes An analyzer

NetSTAT Paper (Method) Network fact base Stand alone application that describes network topology and network services Contains interfaces, hosts, and links Represented as a hypergraph Interfaces are nodes, hosts and links are edges This is a formal model, adds benefits: Well defined semantics Supports reasoning and automation Topological properties described in expressive way

NetSTAT Paper (Method) State transition scenario database Contains signatures of attacks Attacks are sequences of states (snapshots) States are described by assertions that return Boolean values Example: i.link.type==”ATM”; Probes Sensors that are strategically placed in a network but are also full blown intrusion detection system Made up of: Filter that only collects data of interest Inference engine contains attack scenarios Decision engine issues response according to information collected in the inference engine, or reports info to the analyzer

NetSTAT Paper (Method) Analyzer Takes as input a network fact base and a state transition scenario Tells security officer where probes are needed Sets up the probes It determines: Events to be monitored, The network topology State information it requires to verify state assertions

NetSTAT Paper (Architecture)

Analysis: Yacinsac Advantages Able to find flaws in protocols that get past formal analysis Able to detect flaws in concurrently running protocols Architecture is cheap and versatile Disadvantages How do you choose the sources for signatures? How many signatures is too many? Architecture Every single principal required to run software to report to central authority Intruders can disable software Network attacks can still occur unnoticed

Analysis: Pouzol Advantages Allows security officer to specify an equivalence relation to prevent choking attacks on the IDS Formal specification of signatures and detection rules proven sound and complete Disadvantages Has not been implemented in any IDS Complexity of algorithm may create choking attacks Equivalence relations can be dangerous if configured incorrectly

Analysis: NetSTAT Advantages: Can detect intrusions on multiple sub-networks and total network Scalable to large networks Formal methods allow expressiveness and automation Disadvantages Not yet fully implemented Analyzer does ad hoc configuring of probes

Combination Pouzol’s technique to prevent choking attacks can be used by Yasinsac (and NetStat) Two full intrusion detection architecture Which one is best? NetSTAT! Yasinsac’s knowledge base can be used by NetSTAT (and all IDS)

Conclusion Formal methods and intrusion detection can work together to make networks more secure There are many different areas where formal methods can be applied Neither is a silver bullet to network security Attackers are always evolving new techniques to attack a network, and as security experts, so must we

Main References A. Yasinsac. An Environment for Security Protocol Intrusion Detection. Special edition of the Journal of Computer Security, 2001 J. Pouzol and M. Ducassé. Formal Specification of Intrusion Signatures and Detection Rules. 15th IEEE Computer Security Foundations Workshop, June 2002 G. Vigna and R. Kemmerer. NetSTAT: A Network- based Intrusion Detection Approach. Computer Security Applications Conference, 1998