IDS SOEN321, Fall 2004 Serguei Mokhov

Contents IDS intro What it is good for How can you do it (anomaly detection, misuse detection) How it can be compromised (at least, automatic forms of it) Biggest problems.

Intro Started as detection of anomalous network traffic –Difference with the firewalls? (prevention vs. detection) More sophisticated: monitoring behavioral patterns of programs (system calls)/users (typical commands/activities)

Principles –Actions of users and processes are statistically predictable. –Actions of users and processes do not include sequences of commands that subvert security. –Actions of processes lie inside a set of actions permitted by the security policy. Violation of any of the above is an indicator of an attack.

Example An attacker wants to install a backdoor. He enters as an ordinary user and then becomes root. This is unusual if the user is not part of the admin group (principle 1). While becoming root, the attacker uses "evil" sequences of commands (principle 2). Moreover, any changes to system files may cause them to behave in ways they are not supposed to (principle 3). Also, modifying a user file may allow processes executing on behalf of that user to, say, connect to sites they were unable to connect to before or by executing commands they could/did not execute before (principle 1). Note: Intrusions can be arbitrarily sophisticated.

Rootkits and Attack Tools Definition: An attack tool is an automated tool designed to violate a security policy. Example: 'rootkit' is an attack tool; among other things, it sniffs passwords. It also installs fake versions of system programs. These include fake: –netstat: conceals certain network connections –ps: conceals certain processes (e.g., the sniffer) –ls: conceals certain files (as does 'du') ifconfig: conceals promiscuous mode –login: accepts "magic" password The cryptographic checksums of the fake programs are perfect. We see that most of the obvious traces of an attack have been hidden. But, there are still certain integrity checks: –used blocks plus free blocks = all blocks –'ls' should tell the truth about file counts –load average should reflect number of running processes We can assess these quantities by programs other than the fake ones and do the analysis. 'Rootkit' did not corrupt the kernel/file structure, so other programs will continue to report correct information; only the fake programs lie.

Tripwire File modification checks Good against rootkits

Automation Can we automate the intrusion-detection process? We want to automate: - anomaly detection - misuse detection - specification-based detection

Goals of an IDS Detect a wide variety of intrusions (including from insiders). Detect intrusions in a timely fashion (both in real and non real time). Digest/abstract. Present summary to expert human. Accuracy (false positives, negatives). Generally –anomaly models are statistically based –misuse models are signature based –specification models are specification based

Anomaly Detection Anomaly detection requires adaptive statistical profiling. There is an infinite range of possibilities if only because there is an infinite range of variables. Suppose I'm a mole and I want to read sensitive files without attracting attention. My daily average is to peruse 5 files a day. So, I plan ahead, opening 6, then 7, then 8,... until my profile will not be disturbed as I forage widely in files outside my "need-to-know" range. Urban legend: Nervous people are shaky (they emit "tells") when they use touchtone phones. Back-of-the-envelope absurdity, if untargeted. But, there is biometrics, typing rhythms, pseudopolygraphs. Formally, this requires either pure math (some form of statistics) or AI (machine learning of predictive models. Note that data mining is a generalization of statistics so many, many things are possible in principle.

Misuse Modeling Signature analysis actually generalizes to rule-based detection, but this requires a knowledge of system vulnerabilities. This gets us into different flavors of AI. One stab at a distinction: –Does the sequence of data match any of the rules (of bad stuff) as opposed to is it kind of unusual? There is some brittleness here because you first have to have a good set of rules. If anomaly detection is the art of looking for unusual states, then misuse detection is the art of looking for states known to be bad. (Admins can tweak the rules, say in Network Flight Recoder, NFR.)

Agent Who gathers the information? Obviously, it needs to be filtered (downselected) before it can be analyzed. –host-based information gathering: system and application logs –network-based information gathering: monitor network traffic ( tcpdump ), detect network-oriented attacks, use network sniffing The major subtlety is that the analysis software must ensure that the view of the network traffic is identical across the set {analyzer, host1, host2,...}.

Notifier Automatic or incident response. Automatic: If the IDSs on fw1 and fw2 detect a coordinated attack, they can instruct fw3 to reject packets from the source(s) of the attacks. Let's look at an example of an IDS for detecting network intruders in real time. - passively monitor network link over which an intruder's traffic passes - filter the network-traffic stream into a sequence of higher-level events - feed events into an analysis engine that checks conformity with the security policy We will also - look at a number of attacks that attempt to subvert passive monitoring systems fw1 fw2

Design Goals for a Network Monitor no packet-filter drops: the missing traffic might contain precisely the interesting traffic; after all, an attacker might attack the monitor itself real-time notification extensible to include knowledge of new types of attacks

Attacks on the monitor overload, crash, subterfuge: –Overload: Drive the monitor to the point of overload, then attempt a network intrusion. Defense: Can be mysterious about how much load you can handle. –Crash attacks: Kill the monitor (failure, resource exhaustion), then attack as before. Defense: not particularly interesting. –Subterfuge attacks: The key idea is to find a traffic pattern that is interpreted in one way by the monitor and by an entirely different way by the receiving endpoint (target host). Example: What if the monitor sees a particular packet that the endpoint does not? How could this be accomplished by the attacker? Launch a packet with an IP "Time to Live" (TTL) field just sufficient to get it past the monitor but not sufficient to reach the endpoint. Or, suppose the endpoint has a smaller "Maximum Transmission Unit" (MTU). Just send a packet that is too big with the "Do not fragment" bit set.

Signature analysis is when the IDS is programmed to interpret a certain series of packets, or a certain piece of data contained in a packet, as an attack. Example: –An IDS that watches web servers might be programmed to look for the string "phf" as an indicator of a CGI-script attack. Most signature analysis is simple pattern matching: Find "phf" in "GET/cgi-bin/phf?“ –Let's look at insertion and evasion attacks.

Insertion attack Here, an IDS accepts a packet that the end system rejects. Thus, the IDS alone might see the "funny" packets. This defeats signature analysis because it defeats the pattern matching. To illustrate, consider: GET/cgi-bin/....p....h....f....? where the dots stand for funny packets that only the monitor sees.

Evasion attack Here, the end-system accepts a packet that the IDS rejects. There are all kinds of reasons why one computer can be more or less strict about accepting packets than another (checksums, fragments, sequence numbers,...). Since the packets are rejected by the monitor, the attack is not seen.

Conceptual basis IDS either - generates a model of a program's or system's response to known inputs, or - requires the generation of a rule base An IDS that monitors raises an alarm if it detects a deviation from the model or a rule fires. But how can you machine-learn the model? We need some human help. Expert Systems?

Cont’ Can we detect when an application has been penetrated and is then exploited to do harm to other parts of the system? app | frame IDS problem as a sandbox problem | v specify allowed sequences of system calls O S Think of the (uncorrupted) application as a finite-state machine (transition system) whose outputs are sequences of system calls. A transition system can only generate _certain_ sequences of calls. If you observe an impossible call sequence, it is likely that an attacker has introduced malicious code into the application's thread of control by means of a buffer overflow, a format-string attack, or whatever.

Cont’ We detect the insertion of malicious code by observing sequences of system calls that normally could not have occurred. We use human intelligence to build a very abstract model of the application we wish to monitor. We quickly detect "exploit code" running with the application's privilege. We first build a very abstract model of the application's (normal) behavior. Most existing exploit scripts grab full root privilege and take other distinctive actions, such as launching a shell under the attacker's control. This is so blatant it does not require a sophisticated IDS to detect it. But this abstract-model approach is able to detect when some of the backdoors (fake programs) of 'rootkit' are executed, which causes the behavior to deviate from that specified by the original source code and captured in the abstract model.

Problems with IDS Biggest: false positives Require high level of expertise Attacks (previously mentioned can be conducted).

IDS vs. Firewall Would you choose one of if you can afford only one of them: –IDS –Firewall If you can afford both, would you opt for IDS? Recall: firewalls prevent what IDSs detect.

Refs These slides largely based on Dr. Probst transcripts and the textbook.