1. A Survey of Host Based Intrusion Detection Systems (HIDS) Emre Can Sezer Dept. of Comp. Science North Carolina State University

2. 2 Outline Introduction and Motivation Model Creation Techniques Sampling of Models: –N-gram model –Callgraph, Abstract Stack models Impossible Path Exploit Brief overview of VtPath, Dyck and VPStatic Data attacks Conclusion

3. 3 Introduction Terminology –IDS: Intrusion detection system –IPS: Intrusion prevention system –HIDS/NIDS: Host/Network Based IDS Anomaly vs. Intrusion Detection –Anomaly also captures misuse : There is no intrusion, however, due to bad programming or administering, the process behaves differently than normal (i.e. a bug in the code) Intrusions are also anomalies Difference between IDS and IPS –Detection happens after the attack is conducted (i.e. the memory is already corrupted due to a buffer overflow attack) –Prevention stops the attack before it reaches the system (i.e. shield does packet filtering)

4. 4 Introduction Cont’ Idea behind HIDS –Define normal behavior for a process Create a model that captures the behavior of a program during normal execution. –Monitor the process Raise a flag if the program behaves abnormally

5. 5 Why System Calls? (Motivation) The program is a layer between user inputs and the operating system A compromised program cannot cause significant damage to the underlying system without using system calls i.e Creating a new process, accessing a file etc.

6. 6 Model Creation Techniques Models are created using two different methods: –Training: The programs behavior is captured during a training period, in which, there is assumed to be no attacks. Another way is to craft synthetic inputs to simulate normal operation. –Static analysis: The information required by the model is extracted either from source code or binary code by means of static analysis. Training is easy, however, the model may miss some of the behavior and therefore produce false positives. Static analysis based models produce no false positives, yet dynamic libraries and source code availability pose problems.

7. 7 Definitions for Model Analysis If a model is training based, it is possible that not every normal sequence is in the database. This results in some normal sequences being flagged as intrusions. This is called a false positive. If a model fails to flag an intrusion, this is called a false negative. Accuracy: An accurate model has few or no false positives. Completeness: A complete model has no false negatives. Convergence Rate: The amount of training required for the model to reach a certain accuracy

8. 8 A Visual Description of False Positives and Completeness Normal Behavior Model

9. 9 A Visual Description of False Positives and Completeness False Positives Normal Behavior Model

10. 10 A Visual Description of False Positives and Completeness Normal Behavior Model False Negatives

11. 11 N-Gram Pioneering work in the field. Forrest et. al. A Sense of Self for Unix Processes, 1996. Tries to define a normal behavior for a process by using sequences of system calls. As the name of their paper implies, they show that fixed length short sequences of system calls are distinguishing among applications. For every application a model is constructed and at runtime the process is monitored for compliance with the model. Definition: The list of system calls issued by a program for the duration of it’s execution is called a system call trace.

12. 12 N-Gram: Building the Model by Training Slide a window of length N over a given system call trace and extract unique sequences of system calls. Example: System Call trace Unique Sequences Database

13. 13 N-Gram: Monitoring Monitoring –A window is slid across the system call trace as the program issues them, and the sequence is searched in the database. –If the sequence is in the database then the issued system call is valid. –If not, then the system call sequence is either an intrusion or a normal operation that was not observed during training (false positive) !!

14. 14 Experimental Results for N-Gram Databases for different processes with different window sizes are constructed A normal sendmail system call trace obtained from a user session is tested against all processes databases. The table shows that sendmail’s sequences are unique to sendmail and are considered as anomalous by other models. The table shows the number of mismatched sequences and their percentage wrt the total number of subsequences in the user session

15. 15 Problems with Sequence Based Approaches The minimal foreign sequence problem An attack sequence S0,S3,S4,S2 cannot be detected Database includes: … S0,S3,S4 S3,S4,S2 …

16. 16 Problems with Sequence Based Approaches Cont’ Code insertion: –As long as the order in which an attacker issues system calls are accepted as normal, he can insert and run his code on the system (i.e. buffer overflow)

17. 17 FSA Model Sekar et. al., A Fast Automaton-Based Method for Detecting Anomalous Program Behaviors, 2001. Build a non-deterministic finite state automata (FSA) by training. Uses program counter (PC) information to address code insertion problems. Once PC is coupled with system calls, every system call site in the code becomes unique. Instead of using sequences and be limited by length, they use finite state automaton to express every possible sequence. The first piece of research to use PC information and automata.

18. 18 FSA Example An example code and the corresponding FSA built from it S0,S3,S4,S2 is captured. No length limitation. Note the non-determinism in states 1,3,6 and 8.

19. 19 Convergence Comparison Experiment is run on ftpd. FSA model converges faster than N-gram.

20. 20 Callgraph and Abstract Stack Models Wagner et. al., Intrusion Detection via Static Analysis, 2001. Uses finite state automaton to model the process behavior. It is based on static analysis of source code. They introduce three methods: –Callgraph (NFA) –Abstract Stack (PDA) –Digraph (a static version of N-gram with window size of 2, not mentioned here)

21. 21 Callgraph Model A control flow graph (CFG) is extracted from the source code by static analysis. Every procedure f has an entry(f) and exit(f) state. At this point the graph is disconnected. It is assumed that there is exactly one system call between nodes in the automaton and these system calls are represented by edges. Every function call site v, calling f, is split into two nodes v and v’. Epsilon edges are added from v to entry(f) and from exit(f) to v’. The result is a single, connected, big graph.

22. 22 Callgraph Example Entry point Function call site is split into two nodes Epsilon edges Entry(g) v w Exit(g) open() close() exit() Entry(f) Exit(f) getuid() geteuid()

23. 23 Monitoring Callgraph The IDS is given system call information alone, and no PC information. When a system call is received, the automaton is simulated to transition between states. If such a transition does not exist in the model, the IDS raises a flag. Due to non-determinism, there might be more than one possible state at a given time. In this case every possible state in the program is simulated against the system call and the ones that do not have a transition on the given system call are dropped. Non-determinism usually incurs too much computational overhead in large programs.

24. 24 Imprecision in Callgraph Valid Path Impossible Path. Yet the model will not be able to detect it since all transitions are valid. The return address in f can be overridden.

25. 25 Abstract Stack Model The more information an IDS has, the more accurately it can model the behavior of a program. Abstract Stack model makes use of the call stack. In order to incorporate this information into their model, they use a push-down automata (PDA). The idea is to have an abstract copy of the call stack in the PDA stack. At any given state, the PDA’s stack contains the list of return addresses in the call stack.

26. 26 Push-down automata As in FSA, PDA have a set of states and a transition function. They differ from FSA by also having a stack. They accept context-free languages. At every transition, a symbol can be pushed or popped from the stack. They can accept either by state or by stack (if stack is empty), which are equivalent in terms of computational power. PDA is stronger than FSA. It can accept regular languages and also some irregular ones such as 0 n 1 n. StartEnd 1 push 0 0 1 pop 0 Once you see a 1, switch to the End state. The stack contains as many 0 as seen in the input. If the stack is empty at the end of the input, accept. Stack

27. 27 Detecting the IPE Attack Consider the previous example of an impossible path. The Abstract Stack model will detect the attack since it stores stack information. When returning from state Exit(f), the stack will have the return address v’. State v’ does not have a transition on system call exit() hence the attack will be detected.

28. 28 Performance Issues Both of the models Callgraph and Abstact Stack have very high operational costs. The reason for this is non-determinism. Non-determinism manifest itself in two ways: –State non-determinism: The automaton can be in a number of different states. When a system call is received, all these states need to be checked for valid transitions. –Stack non-determinism: Only applies to Abstract Stack model. There can be a number of different ways a state can be reached, resulting in more than one stack configuration.

29. 29 State and Stack Exposure Techniques Exposing state can greatly reduce the non-determinism in the model. The state of the program can be exposed by using PC information. The stack can be exposed in two ways: –Indirectly as in Abstract Stack, where the PDA has transitions that simulate the call stack. –Directly by stack walk, simply obtaining the list of return addresses from the call stack.

30. 30 VtPath Feng et. al., Anomaly Detection Using Call Stack Information, …. Inspired by Abstract Stack, they use call stack information in their model. It is training based and has better convergence rate and comparable false positive rates than the FSA model. Uses virtual stack lists to create virtual paths between two consecutive system calls and keeps a database of these virtual paths. It uses PC information and stack walk to get the VSL’s. The model is a collection of virtual paths. More resistant to IPE’s. It can capture the IPE presented in Wagner et. al.’s paper.

31. 31 Dyck Giffen et. al., Efficient Contest-Sensitive Intrusion Detection, 2004. Uses static analysis of binary code. Exposes stack by inserting null-calls before and after function call sites. Null-calls are inserted before and after function call sites to keep track of function calls using binary rewriting. With null-calls, the stack becomes deterministic and the performance improves greatly compared to a non- deterministic PDA. This model is called a stack-deterministic PDA (SDPDA) in a later paper by the same authors. Feng et. al., Formalizing Sensitivity in Static Analysis for Intrusion Detection, 2004.

32. 32 Dyck: Model Example C source code exapmle Dyck instrumentation

33. 33 Dyck: Model Example Cont’ Callgraph Model Dyck Model w/o Squelching

34. 34 VPStatic Feng et. al., Formalizing Sensitivity in Static Analysis for Intrusion Detection, 2004. Static analysis version of VtPath. Instead of using sequences, they define transitions on a PDA. The state of the program is exposed by using PC information. The stack is exposed by using stack walk and VSL’s. The goal is to create a deterministic PDA by exposing stack and state information. The model is fully deterministic. Operating the deterministic PDA is less expensive, however, the bottleneck in VPStatic is the stack walk operation.

35. 35 Overview of the Models The trend has been towards more complicated automata and static analysis. Models using state exposure are immune to code insertion. i.e FSA, VtPath, VPStatic. Models using stack exposure are immune to control-flow hijacking. i.e Abstract Stack, Dyck, VPStatic. Still, if an attack does not issue system calls, these models might fail.

36. 36 Data Flow Attack A variation of the IPE. The control flow is altered but not hijacked. Instead of overwriting return addresses to change the control flow, a data used as a predicate in a branch is overwritten. The Data Flow Attack does not traverse any function boundaries evading even PDA based models. The models need to be flow sensitive in order to capture such an attack.

37. 37 Data Flow Attack Example The system call sequences and are normal sequences. Any of the afore mentioned models using will also accept and There is no way the model can relate the first loop to the second. Execution path history needs to be known to be able to detect such an attack. sys_5 ();

38. 38 User ID Hijacking Example attack on WU-FTPD. When a user issues a get or a put command, the effective user id (EUID) is temporarily escaladed to root in order to perform setsockopt(). Using format string vulnerability, pw->pw_uid can be set to 0 (root), giving root privileges to the user. FILE * getdatasock(... ) {... seteuid(0); setsockopt(... );... seteuid(pw->pw_uid);... }

39. 39 Decision-Making Data Hijacking The following example code is taken from a SSH implementation. The function detect_attack() has an integer overflow vulnerability. Using the vulnerability, the authenticated flag can be set to non-zero, allowing a user root privilege without him ever supplying a password. void do_authentication(char *user,...) { 1: int authenticated = 0;... 2: while (!authenticated) { /* Get a packet from the client */ 3: type = packet_read(); // calls detect_attack() internally 4: switch (type) {... 5: case SSH_CMSG_AUTH_PASSWORD: 6: if (auth_password(user, password)) 7: authenticated =1; case... } 8: if (authenticated) break; } /* Perform session preparation. */ 9: do_authenticated(pw); }

40. 40 Why Automata Can’t Capture Data Flow Attack With the call to the system call in between the branches (sys_5), the model looses all execution path information. None of the models mentioned are neither flow nor path sensitive. Start Sys_ 1 Sys_ 2 Sys_ 5 Sys_ 3 Sys_ 4 End There are no function calls, so stack exposure is ineffective against this attack. In the absence of function calls, all the models keep track of consecutive system calls. In other words, they are only as powerful as N-gram with a window size of 2. Normal path Abnormal path

41. 41 A Different Look at System-Call Based IDS’s The problem is recording every possible system call trace an application can produce. In doing so, other security issues such as code injection and mimicry attacks must be considered. The models we have seen are compact approximations for these infinite sets.

42. 42 Back To Training Based Models The data attack can be detected in two ways: –Finer grained methods: Live analysis of variables, or checking predicates at branches. –Using training: Normal user sessions will not produce sequences seen with the data attack. Using finer grained models beats the purpose of having system-call based IDS’s.

43. 43 Execution Path History Given a node v in the graph, the execution path will go through a number of branch instructions and loops before reaching this state. If we were able to keep track of the execution path that was taken up to node v, we could append that information to the node using training. In the data attack example, node Sys_3 would know that only an execution path that’s been through node Sys_1 should exist. Start Sys_ 1 Sys_ 2 Sys_ 5 Sys_ 3 Sys_ 4 End Start, Sys_1, Sys_5 Start, Sys_2, Sys_5

44. 44 Obtaining Execution Path History One major tool in accomplishing this task could be null-call insertion. –It is used in the Dyck model to keep track of function call sites. The idea can be applied to every branch that issues a system call. A sequence of previously issued system calls can be appended to every node. During monitoring, the execution path history will be matched against the possible histories at every node.

45. 45 Performance Considerations Every node will have a great number of possible histories that needs to be kept track of. Considering the size of today's applications, recording a list of these paths for every node is clearly not possible. Compact representations must be developed. For example, a node that has only a single incoming edge needs not keep an entire record of histories as its history is a single system call appended to its predecessors execution path history. Also not every node on the path is critical. If most of the execution paths have common substrings, there might be a way to extract the important information from the sequence.

46. 46 Conclusion When it comes to real time intrusion detection, false positives are unacceptable. This has lead researchers towards static analysis based, complicated models such as Dyck and VPStatic. Yet, the data attacks shows that even these models are not complete. Still, these models should not be underrated, since they can capture code insertion, stack corruption and impossible path attacks with no false positives.