CSC 382/582: Computer SecuritySlide #1 CSC 382/582: Computer Security Integrity Management

CSC 382/582: Computer SecuritySlide #2 Topics 1.Host Integrity 2.Anti-virus Software –Malware self-protection techniques. –Malware detection techniques. 3.Personal Firewalls 4.Host Intrusion Detection Systems 5.Host Intrusion Prevention Systems

CSC 382/582: Computer SecuritySlide #3 Host Integrity Ensuring a host adheres to a security policy. C AI 1.Misuse Detection and Prevention –Know a set of attacks against a host. –Attempt to detect and prevent these attacks. –Anti-virus, anti-spyware, personal firewall. 2.Policy-based Tools –Security policy describes good state of system. –Attempt to detect deviations from good state. –Host Intrusion Detection Systems (HIDS).

CSC 382/582: Computer SecuritySlide #4 Misuse Detection Anti-virus tools most widely used host misuse detection software. –Signature database primary method of detecting known attacks on host. Most anti-virus tools also offer –Intrusion Prevention: detect virus during d/l. –Recovery: eradicate detected viruses. Anti-virus software is often deployed on network proxy servers in addition to hosts.

CSC 382/582: Computer SecuritySlide #5 Malware Self-Protection Anti-debugging Detect/disable debuggers when used to analyze code. Attack anti-malware tools Disable anti-malware tools upon infection. Kill processes or destroy/modify signatures. API checksums Avoid having UNIX/Win32 API calls in code. Store checksums of API names and search for match. Code obfuscation Use unusual tricks and unused code to avoid dissassembly and prevent quick analysis of purpose. Self-modifying code.

CSC 382/582: Computer SecuritySlide #6 Self-Protection Compression Code looks almost random; size is smaller. Use unusual executable packers to avoid analysis. Data encryption Encrypt strings, hostnames, IP addresses to avoid detection. Embedding Embed infection in one format inside a document inside an archive file. Scanners have to understand and have time to parse and decompress each file format.

CSC 382/582: Computer SecuritySlide #7 Self-Protection Entry-Point Obscuring Changing initial code or entry point easy to notice. Alter program code to gain control randomly. Host morphing Alter host file during infection to prevent removal.

CSC 382/582: Computer SecuritySlide #8 Self-Protection: Encryption Encrypt all code except small decryptor. –Note that copy protected files will have similar decryptors to prevent analysis too. –Often uses multiple decryptors. –Change encryption key dynamically. Random Decryption Algorithm (RDA) –Choose random key for encryption. –Brute force search for key to decrypt. –Slows VMs/debuggers used for analysis.

CSC 382/582: Computer SecuritySlide #9 Self-Protection: Polymorphism Alter malware code with each infection. –Cannot be detected by signature scanning. –May alter decryptor only or entire code. –Insert junk instructions that do nothing. –Fragment and rearrange order of code. –Alternate sets of instructions for the same task. Ex: SUB -1 instead of ADD 1 –Randomize names in macro viruses.

CSC 382/582: Computer SecuritySlide #10 Case Study: Zmist EPO, encrypted, polymorphic virus. Code integration Decompiles PE files to smallest elements. Inserts virus randomly into existing code. Rebuilds executable. Polymorphic decryptor Inserted as random fragments linked by JMPs. Randomizes self with ETG engine.

CSC 382/582: Computer SecuritySlide #11 Virus Detection Signature-based –Look for known patterns in malicious code. –Defeated by polymorphic viruses. Smart scanning –Skips junk instructions inserted by poly engines. –Skips whitespace/case changes in macro viruses. Decryption –Brute-forces simple XOR-based encryption. –Checks decrypted text against small virus sig to decide whether has plaintext or not.

CSC 382/582: Computer SecuritySlide #12 Virus Detection Code Emulation –Execute potential malware on VM. –Scan VM memory after certain # iterations. –Watch instructions for decryptor profile. Code Optimization. –Optimize away junk instructions and odd techniques used by polymorphic viruses.

CSC 382/582: Computer SecuritySlide #13 Virus Detection Heuristics –Code execution starts in last section. –Suspicious code redirection. –Suspicious section ACLs or size. –Suspicious library routine imports. –Hard-coded pointers into OS kernel. Neural Network Heuristics –IBM researchers trained neural net to recognize difficult polymorphic viruses. –Released in Symantec antivirus.

CSC 382/582: Computer SecuritySlide #14 Limits of Malware Detection Assume you have a perfect malware detector D(p) that takes a program p as input and returns True or False. Create a program P(q) that incorporates your malware detector D : if D(q): Do nothing else Become malware What would D report if given P to analyze?

CSC 382/582: Computer SecuritySlide #15 Personal Firewall Firewall configured to protect single host. –Used on servers, desktops, and laptops. Why use a personal firewall? –Configuration can closely match single host’s needs without considering entire network. –Can protect on a per-application basis. –Can protect mobile hosts when outside the organization’s network firewall.

CSC 382/582: Computer SecuritySlide #16 Host Intrusion Detection Systems Monitors host state for signs of intrusion: 1.Files File metadata changes (access time, perms) File data changes (checksums) 2.Configuration Log file entries User accounts and groups 3.Runtime Logins and logouts Running processes Open network connections Kernel modules and status

CSC 382/582: Computer SecuritySlide #17 Why HIDS? October 25, 2000 –Microsoft detects passwords being mailed out of company to address in Russia. –Electronic logs show source code downloaded. How did it start? –Employee received carrying Qaz trojan. –Qaz copied itself to Notepad.exe, moved Notepad to Note.com. –Qaz spread itself across network, d/led tools. –Attacker used tools to acquire passwords.

CSC 382/582: Computer SecuritySlide #18 Advantages of HIDS 1.HIDS can associate data with specific users, while NIDS cannot do this. 2.HIDS has access to data that’s sent to the host in a network encrypted form (SSL, VPN.) 3.HIDS is immune to NIDS evasion techniques. NIDS doesn’t know how host stack interprets packets, but HIDS looks at data after stack has processed it.

CSC 382/582: Computer SecuritySlide #19 HIDS Types 1.File Integrity Checkers –Tripwire, AIDE, anti-virus software 2.Log Watchers –logwatch, swatch 3.Network monitors –portsentry, BlackICE 4.Host Integrity Monitors –Osiris, Samhain

CSC 382/582: Computer SecuritySlide #20 File Integrity Checkers 1.Perform baseline scan of filesystem. –Metadata: ownership, permissions, times –Cryptographic checksums of contents 2.Periodically scan filesystem –Compare current state to baseline state. –Notify admin if changes discovered.

CSC 382/582: Computer SecuritySlide #21 File Integrity Checkers Configuration required to avoid false +’s –Ignore temporary files in /tmp and elsewhere. –Ignore log file checksums, but permissions and ownerships are important and size should not decrease. –Update checksums when binaries or libraries updated.

CSC 382/582: Computer SecuritySlide #22 HIDS Architecture A HIDS consits of three components: –Agent runs on host and gathers data. –Director periodically polls hosts and aggregates data Sends requests to agents to scan hosts. Receives data from agents. Decides on whether to act on data. –Notifier acts on director results. May simply notify security officer. May reconfigure agents or director. May activate response mechanism.

CSC 382/582: Computer SecuritySlide #23 Agent Scans host and transmits data. Attackers target agent to avoid detection –Kill agent process. –Replace agent with their own code. –Install rootkit. Protection against subversion –Self integrity check. –Privilege separation. –Encrypted communication with director. –Configuration data and baselines stored on director. –Scan data transmitted to director host only.

CSC 382/582: Computer SecuritySlide #24 Director Director Functionality 1.Polls agents to request scan data. 2.Sends configuration data to agents. 3.Receives scan data from agents. 4.Stores scans in database. 5.Compares scans with baselines. 6.Updates baselines if Administrator requests. Protecting the Director –Use a dedicated host with limited access. –Run an agent on the director host to monitor itself.

CSC 382/582: Computer SecuritySlide #25 Example HIDS: Tripwire

CSC 382/582: Computer SecuritySlide #26 Running a HIDS When should I collect baselines? –Ideally before the host is connected to network. –If later, knowngoods.org maintains checksums of common Linux distributions. How often should I poll hosts? –Depends on performance goals. How often should I monitor alerts? –Serious alerts should automatically notify you. –Read logs daily.

CSC 382/582: Computer SecuritySlide #27 Host Intrusion Prevention Systems Attacks usually cause programs to initiate new behaviors: –New network sockets opened. –New files modified, etc. Create a system call model of a program. –Compiler extracts call graph model of program. –HIPS monitors program during execution, checking to see if execution matches model.

CSC 382/582: Computer SecuritySlide #28 Example: System Call Models

CSC 382/582: Computer SecuritySlide #29 Key Points 1.Integrity Management Approaches –Misuse detection and prevention. –Policy-based management. 2.Detection Issues –Problems: encryption, polymorphism –Solutions: compiler-based, VM simulations 3.Host Integrity Management Systems –System Types: HIDS, HIPS –Monitor Files, Configuration, Runtime data. –HIDS provides local view with greater detail than NIDS.

CSC 382/582: Computer SecuritySlide #30 References 1.Matt Bishop, Computer Security: Art and Science, Addison-Wesley, Ted Bridis and Rebecca Brucman, “Microsoft hacked! Code stolen?”, October 26, Simson Garfinkel, Gene Spafford, and Alan Schartz, Practical UNIX and Internet Security, 3/e, O’Reilly & Associates, Lap Chung Lam, Wei Li, and Tzi-cker Chiueh, “Accurate and Automated System Call Policy-Based Intrusion Prevention,” Proceedings of the International Conference on Dependable Systems and Networks (DSN'06), Stephen Northcutt, Lenny Zeltser, Scott Winters, Karen Kent and Ronald Ritchey, Inside Network Perimeter Security, Second Edition, Sams Publishing, Ed Skoudis and Lenny Zeltser, Malware: Fighting Malicious Code, Prentice Hall, Ed Skoudis, Counter Hack Reloaded 2/e, Prentice Hall, Peter Szor, The Art of Computer Virus Research and Defense, Addison- Wesley, Brian Wotring, Host Integrity Monitoring Using Osiris and Samhain, Syngress, 2005.