CSC 382/582: Computer Security Integrity Management CSC 382/582: Computer Security

Models of Intrusion Detection Misuse detection You know what’s bad. Attempt to detect bad items. Anti-virus and anti-spyware tools. Anomaly detection You know what’s good. Attempt to detect deviations from good state. Host Intrusion Detection Systems (HIDS). CSC 382/582: Computer Security

Theory of Malicious Code Theorem 22-1: It is undecidable whether an arbitrary program contains a computer virus. Proof: Define virus v as TM program that copies v to other parts of the tape, while not overwriting any part of v. Reduce to Halting Problem: T’ running code V’ reproduces V iff running T on V halts. Theorem 22-2: It is undecidable whether an arbitrary program contains malicious logic. CSC 382/582: Computer Security

CSC 382/582: Computer Security Detecting Malware Signature-based Look for known patterns in malicious code. Defeated by polymorphic viruses. Smart scanning Skips junk instructions inserted by poly engines. Skips whitespace/case changes in macro viruses. Decryption Brute-forces simple XOR-based encryption. Checks decrypted text against small virus sig to decide whether has plaintext or not. CSC 382/582: Computer Security

CSC 382/582: Computer Security Detecting Malware Code Emulation Execute potential malware on VM. Scan VM memory after certain # iterations. Watch instructions for decryptor profile. Code Optimization. Optimize away junk instructions and odd techniques used by polymorphic viruses. CSC 382/582: Computer Security

CSC 382/582: Computer Security Detecting Malware Heuristics Code execution starts in last section. Suspicious code redirection. Suspicious section ACLs or size. Suspicious library routine imports. Hard-coded pointers into OS kernel. Neural Network Heuristics IBM researchers trained neural net to recognize difficult polymorphic viruses. Released in Symantec antivirus. CSC 382/582: Computer Security

CSC 382/582: Computer Security Detecting Malware Behavior-based Watch for known actions from malicious code. Network access signature of worm. Unexpected use of dangerous system calls. Integrity Checking Host-based Intrusion Detection System. Record MAC, size, dates, ACL of files. Periodically check for changes. ex: Tripwire, AIDE CSC 382/582: Computer Security

CSC 382/582: Computer Security References Ross Anderson, Security Engineering, Wiley, 2001. Matt Bishop, Computer Security: Art and Science, Addison-Wesley, 2003. William Cheswick, Steven Bellovin, and Avriel Rubin, Firewalls and Internet Security, 2/e, 2003. Fred Cohen, http://www.all.net/books/virus/part1.html, 1984. Simson Garfinkel, Gene Spafford, and Alan Schartz, Practical UNIX and Internet Security, 3/e, O’Reilly & Associates, 2003. Cyrus Peikari and Anton Chuvakin, Security Warrior, O’Reilly & Associates, 2003. Ed Skoudis and Lenny Zeltser, Malware: Fighting Malicious Code, Prentice Hall, 2003. Ed Skoudis, Counter Hack Reloaded 2/e, Prentice Hall, 2006. Peter Szor, The Art of Computer Virus Research and Defense, Addison-Wesley, 2005. CSC 382/582: Computer Security