Attacking Malicious Code: A Report to the Infosec Research Council Kim Sung-Moo

Contents  What is malicious code?  Defense against malicious code  Current defenses  Promising new defenses  Policy as Achilles’ Heel  Conclusions

What is Malicious Code?  Any code added, changed, or removed from a software system.  To intentionally cause harm or subvert the system’s intended function.

Examples of malicious code  Viruses  attach to host programs and propagate when an infected program executes.  Worms  Particular to networked computers.  Attack to jump from one to another across network.  Trojan Horses  Hide malicious intent inside a host program that appears to do something useful.  Attack Scripts  Exploit security weaknesses like buffer overflows.  Java attack applets  Embedded in Web pages.  Dangerous ActiveX controls

A Growing Problem  Networks are everywhere.  Internet makes an attack easy.  System complexity is rising.  Applications becoming larger, bugs can’t be avoided.  Hard to find malicious code in API.  The use of unsafe programming language (C or C++)  Systems are easily extensible.  Malicious code can slip in systems as an unwanted extension. (scripts, applets, etc.)

Defense against Malicious Code  4 approaches that the host can protect itself.  Analyze the code and reject it if there is the potential that executing it will cause harm.  Rewrite the code before executing it so that it can do no harm.  Monitor the code while its executing and stop it before it does harm.  Audit the code during executing and take policing action if it did some harm.

Code Analysis  Scan a file or do dataflow analysis.  Help finding security-related bugs. (buffer overflow)  Necessarily limited  Because determining if code will misbehave is as hard as the halting problem.

Code Rewriting  Less pervasive but more important.  Inserts extra code to perform dynamic checks that ensure bad things cannot happen.  Ex) array bounds check in Java

Code Monitoring  Traditional approach  Monitor the code while its executing and stop it before it does harm.  Ex) JVM interpreter  Interpreter monitors execution of applets and mediates access to system calls  By examining the execution stack to determine who is issuing the system call request.

Code Auditing  If we want to recover damage, the damage must be properly assessed and addressed.  Create an audit trail that captures program behavior.

Security Principles  The Principle of Least Privilege  A component should be given the minimum access necessary to accomplish its intended task.  The Principle of Minimum Trusted Computing Base (TCB).  The best way to assure that your system is secure is to keep your TCB small and simple.  TCB is the set of hardware and software components

Current Defenses  OS-Based Reference Monitors  Scanning for Known Malicious Code  Code Signing

OS-Based Reference Monitors  Provided by the computer hardware and OS.  Examples  Address translation hardware  Supervisor-mode and user-mode  Timer interrupt  System calls for invoking a trusted software  Disadvantage  Work with a fixed system-call interface and a fixed vocabulary.  Hard to extend interface and vocabulary.  Advantage  Effective for protecting OS resources.

Scanning for Known Malicious Code  Blacklisting matches programs against a database of known virus signatures.  Disadvantage  Weak to unknown malicious codes  Each file must be scanned  Too easy to change malicious code a little.  Advantage  Cheap to implement  Easily find naïve attacks

Code Signing  Authenticate code based on public-key cryptography and digital signatures.  Poor understanding of most people  If a key is stolen, anyone can use the key to sign any code.  We can’t know developer’s bad thought.  Lack of a Public Key Infrastructure (PKI)

Promising New Defenses  Software-Based Reference Monitors  Type-Safe Languages  Proof-Carrying Code

Software-Based Reference Monitors  Alternative to OS-based reference monitors  Idea  Rewrite binary code  Insert extra checks and states on each memory access and control transfer.  Current research  Implementation of Java’s stack inspection security policy

Type-Safe Languages  Type-safe programming language  Java, scheme, ML  Ensure that operations are only applied to values of the appropriate type.  Specify new, abstract types for operations.  Enable offline enforcement through static type checking instead of each time a operation is performed.  Current research  Eliminate more runtime checks (array bounds checks)  Eliminate type-checking machine code

Proof-Carrying Code  Require any untrusted code to come equipped with an explicit, machine-checkable proof  Before executing  Prove that the code respects a given security policy.  Proof checker is very simple.  Can enforce any security policy with proof  Current research  How to integrate compilers and modern theorem provers to produce PCC.

Policy as Achilles’ Heel  Policy can be of service.  But there is the problem of policy.  Bad policy  Incorrectly enforced policy  There is the many levels of policy.

Bad Policy  Misunderstanding of context  policy makes no sense in the context where it was applied.  Over restriction  the policy prevents useful work when it is enforced.  Noncomprehensiveness  policy fails to cover some situation or exists at the wrong level of abstraction.

Incorrectly Enforced Policy  The enforcement mechanism is too weak to implement the desired policy.  There are bugs in the implementation of the enforcement mechanism.  The enforcement mechanism is misconfigured.

Example Bad PolicyExample Incorrect policy enforcement Example Context misunderstood Script languages (Visual Basic) Mechanism is too weak Privacy in smart cards Overly restrictive Changing passwords too frequently Mechanism is buggy Buffer overflows in kernels Noncompre- hensive Executable content in (love virus) Mechanism is misconfigured Sendmail debug mode

The Many Levels of Policy  By collection of lower-level enforcement mechanisms, define metalevel policies.  We must consider safety, liveness, confidentiality.  Most of technologies can enforce particular aspects of software behavior.  Question  How to express reasonable security policy that can be directly transformed into solutions.  Answer  Understand policy as a layered set of abstractions.

Conclusions  The malicious code problem will continue to grow as the Internet grows.  Our hope is  creating sound policy about software behavior  enforcing that policy through the use of technologies