1. 1 Part 6: System Security u Malware u Safe Coding u Software Trust u Virus Detectors u Software Signatures u “Kernel Integrity Checkers” u “Application Integrity Checkers”

2. 2 Computer Systems u Security – keep the systems safe u No malware u No applications that can be attacked u No data stolen u No corruption or harmful activity u User accounts are not hacked u ….. HOW? Safe software Virus checks Sandboxing Integrity checking Crypto File Systems Smartcards

3. 3 Safe Coding u Not well understood u Software has vulnerabilities, ensure attack free code at implementation time  Stack Guard and Lib Safe and a host of techniques are available  Type safe languages are available  Not clear if these approaches work u Teach programmers to be aware of attack techniques on software  Safe coding practices

4. 4 Bounds Checking u Array bounds often get exceeded and yet nothing happens  Very common in C and C++ code  Optimized compilers for all languages disable bounds checking  Have to be checked in software, at all points u Buffer overflow of many kinds use this u For every data transfer we need to check the size of the data  Function calls  Data copying  Reading input  more

5. 5 Input Validation u Input to a program has to be validated  From network  From files  From user u Almost never done u A program “expects” the data to be formatted correctly, specially if that program generated the data  Suppose WORD creates a DOC file  When WORD reads it, the data is expected to be in WORD format  Simple checks are NOT enough u This vulnerability is ingrained into all legacy software

6. 6 Calling Routines u When routines are called, the caller and the callee must perform sanity checks on arguments u Specially when system calls and library routines are called u Many stack and heap smashing attacks use such vulnerabilities  If a program has a call to the “system” routine in Unix  If arguments are tampered with, any program can be run by an attacker

7. 7 Root Access Attacks u “set uid” programs can gain root access  Used heavily for Unix  Similar features in all operating systems u Many attacks have been designed to misuse this feature in many non-intuitive ways u Databases are particularly attackable via setuid attacks

8. 8 Heap tricks u Put crafted data on heap  Most input is read into the heap area anyway u Ensure that a particular pointer to an argument to another function call has been tampered with  Next time the function is called, it will be called with attacked arguments u Hard to detect and protect against

9. 9 Open Source vs. Closed Source u What is safer – Open Source or Closed Source? u Open Source:  Many eyes theory  When found, vulnerabilities quickly fixed u Closed Source:  Stringent code review and coding security policies may be in effect  Even if vulnerabilities exist, they may not be found, as the code is unknown  Vendor may have liability u This debate has no obvious outcome, both have advantages and disadvantages

10. 10 Data Execution u Many attacks inject executable code as data and then transfer control to it u Solution is to ensure data is never executed  Code resides in code segment, data resides in data segment  Code segment must be non-writable  Data segment must be non-executable u Hardware support needed NX or XD bits in page tables  Existed in the old VAX architecture

11. 11 Software Issues u Software is inherently untrustable u Even if source code is visible (e.g. Thompson Attack)  Many techniques for obfuscation  Easy to make software do things other than what the source code seems to do u Programmers can inject things into programs that can be harmful  By design  By accident u How do we know that software is to be trusted?  OS  Applications  Drivers  Downloaded things No good answer

12. 12 Feature Creep u Features of a program causes unforeseen vulnerabilities u e.g. Visual Basic Scripting in Outlook  Why does an email reader have to be programmable?  ….because it can be done! u Complex features of programs are attractive to designers and to end users u Impossible to analyze the security impacts u Many tricks are then invented by attackers u Old adage: Keep it simple  Code will be cleaner  Features would be understandable  Unforeseen consequences will be minimized

13. 13 Malware u Virus – Trojan – Worm u Adware, spyware u Rootkits Can be detectable, but only if detector in operating system Not detectable

14. 14 Virus Detectors u Current state of the art in virus prevention is detection of viruses in all incoming files and existing files u Any file that a computer accesses is “scanned” by a virus detector u Hooks to the “open” and other system calls  Makes system performance degrade u Scanning via hash matching u Needs to know all the hashes of known viruses  Detects known viruses  Does not detect changes to existing programs  Have to frequently download “virus definitions”

15. 15 Downside of Virus Detectors u Can be easily disabled (via buffer overflow, for example) u Rootkits can uninstall it, or disable it  Cannot detect rootkits u Performance is slow u New viruses can spread without fear for some time u The virus detector is itself a virus?  Yes, but a good one  No, but can be, via a little social engineering u Can store hashes of good files, but….  A virus can change the definitions and escape detection

16. 16 Smart Viruses u Polymorphic Viruses u Viruses that attack detectors u Viruses that hide from detectors  Camouflaged  Hidden names

17. 17 Rootkits u Rootkits are kernel patches + processes u Processes may run with Admin/root privileges u Special routines are embedded in the kernel u Many kernel routines may be changed or patched u How to detect  Take checksum of kernel  Take checksum of routines that may change  Look at kernel tables and watch for changes  Check for perturbations

18. 18 Undetecatable u If a rootkit is not worried about rootkit detectors then it is detectable u Rootkits can be designed to be evasive  If we take hashes of the kernel, it will patch the routine that computes the hash  If we track kernel tables, the tracking routine will be patched u A “new” rootkit detector will find an old rootkit  As long as the rootkit writer does not know of the detector and its techniques  A rootkit can patch itself later to become invisible to new rootkit detector  Cat and Mouse game

19. 19 Sandboxing u Run a program with some flags set in the kernel, designating it to be a sandboxed process u Every time this process calls a “system call”  Check whether it is an allowable operation  Check whether the arguments are “allowed”  Interpret the call and ensure it falls under the “safe” category  Policies have to be stated and enforced  …. Loopholes?

20. 20 Integrity Checking u Every executable file should have a pre-computed hash and a signature and an issuer certificate  Hash ensures any changes to the executable is detectable  Signature ensures that no one updates the hash  Issuer certificate ensures the signature is not faked  This information is stored, separately from the file itself (hopefully securely) u Integrity Checking  Check hash and signature every time the file is executed  Check hash occasionally while the program execution is in progress (to detect dynamic corruption) u Problem: Who does the checking?

21. 21 Application Integrity Checking u Assume: Operating System is not compromised u OS does Application Integrity Checking  OS stores hashes and signatures  Unsigned application either not allowed, or signed by OS  Dynamic checking is also done u Ensures freedom from downloaded viruses u Ensures some freedom from buffer overflow attacks  Cannot be completely detected, but the harm caused is limited u Runs only “trusted applications” u Either signed applications or authorized by the user  Help prevention of undetected downloads, spyware and many such malware

22. 22 Kernel Integrity Checking u How to check the OS for rootkits? u Use a hardware device  TCG “Trusted Computing Group” approach  “Co-Pilot” approach u Use a software device  The VMM “Virtual Machine Manager” approach u Concept:  A trusted device has keys and signatures for the operating system components  This device can access the OS and check it for integrity  Possible to detect rootkits, as long as the checker is not compromised

23. 23 Hardware Checking u TPM (trusted platform module) from TCG  A hardware device, that contains keys and hashes for checking integrity of operating systems  Attestation of software and digital signatures possible  An elaborate framework of security modules, protocols and monitoring of applications  Can be used to enforce software licenses u Co-Pilot Project  Hardware on the PCI bus, can take over the computer  Performs memory checks on the OS and interrupt routines  Can detect rootkits, even dynamic ones

24. 24 Software Checking u Run the Operating System as a host on a Virtual Machine Manager u The VMM is hard to compromise  Small, hence free of vulnerabilities (assumption)  OS has to be rootkitted to attack VMM, but the VMM can detect such rootkits u Check OS integrity from the VMM (Terra Project) u Check OS integrity from a separate host operating system called the Security Monitor  ASU project u Work in progress, not tested well yet

25. 25 Secure I/O u All security solutions outlined so far suffers from a spoofing vulnerability u A piece of software can be flagged to be “trusted” by faking human input by a program (virus) u TO ensure spoof freedom there is a need for a secure I/O channel from the core security system to the human user  TCG-TPM, Co-Pilot, Terra --- all are vulnerable u Secure I/O needs separate hardware channel, separate keypad  An SSL connection to a small communicator may be a solution.

26. 26 Crypto File Systems u All files are stored encrypted u Where is the key?  Derived from a passphrase  Everything is “weak” as along the computer is “running” u MS-CFS uses PKI u Advantages  Lost/stolen disks are secure  Proper implementations may disallow some viruses from accessing some files u Diadvantrages  Files are decrypted on the fly and CFS is transparent to users and hance viruses  Attacking the key is possible

27. 27 Current Status u Integrity checking is almost non-existent in deployed software  Hence, all systems are soft against viral attacks  Big security problem,  Device driver signing is used in Windows, quite ineffective u The TCG system is the closest to deployment  Also the weakest system  Has political implications u No other solutions are being deployed   may not be entirely true