Securing Untrusted Code via Compiler-Agnostic Binary Rewriting Richard Wartell, Vishwath Mohan, Dr. Kevin Hamlen, Dr. Zhiqiang Lin The University of Texas at Dallas Supported in part by NSF, AFOSR, and DARPA 1

Software Fault Isolation (SFI) Automatically rewrite binaries to make them safer [Wahbe, Lucco, Anderson, Graham, SOSP 1993] 2 Untrusted code Rewriter Safe code

Software Fault Isolation (SFI) trusted & untrusted modules in common address space Example #1: web browser plug-ins Example #2: trusted system libraries inside untrusted application Goal: protect trusted modules from untrusted ones confine untrusted module behaviors Example: Untrusted modules must obey trusted module interfaces Blocks ROP attacks [Shacham, CCS 2007] 3 eMule.exe kernel32.dll user.dll Trusted Untrusted

Inlined Reference Monitors (IRMs) SFI foundation supports higher-level policies [Abadi, Budiu, Erlingsson, and Ligatti. CCS 2005] Example: IRMs [Schneider, ISS 2000] Enforces powerful policies: program-specific (no other programs affected) light-weight enforcement (minimize context switches) Statefulness Example: Adobe Reader may access the network (to check for updates) and may read my confidential files, but may not access the network after reading my confidential files. 4 reader.exe kernel32.dll user.dll Trusted Untrusted IRM

A Brief History of SFI Wahbe 1 PittSFIeld 3 CFI / SMAC 2 XFI 4 NaCl 5 1: [Wahbe, Lucco, Anderson, and Graham. SOSP 1993] 2: [Abadi, Budiu, Erlingsson, and Ligatti. CCS 2005] 3: [McCamant and Morrisett. USENIX 2006] 4: [Erlingsson, Abadi, Vrable, Budiu, and Necula. SOSDI 2006] 5: [Yee, Sehr, Dardyk, Chen, Muth, Ormandy, Okasaka, Narula, and Fullagar. S&P 2009]

A Brief History of SFI Wahbe 1 RISC only PittSFIeld 3 Special GCC CFI / SMAC 2 Needs PDB XFI 4 Needs PDB NaCl 5 Special GCC All prior works require explicit code-producer cooperation 1: [Wahbe, Lucco, Anderson, and Graham. SOSP 1993] 2: [Abadi, Budiu, Erlingsson, and Ligatti. CCS 2005] 3: [McCamant and Morrisett. USENIX 2006] 4: [Erlingsson, Abadi, Vrable, Budiu, and Necula. SOSDI 2006] 5: [Yee, Sehr, Dardyk, Chen, Muth, Ormandy, Okasaka, Narula, and Fullagar. S&P 2009]

Reins: REwriting and IN-lining System Main Discovery: means of enforcing SFI for near arbitrary COTS binaries no source code or debug info (assumed unavailable) no disassembly listing compiler-agnostic real COTS binary features interleaved code and data computed control-flows dynamic linking event-driven callbacks multithreading Low overhead (~2%) Formal machine-verification of policy enforcement 7

Binary Rewriting w/o metadata Relocation information, debug tables and symbol stores not always available Reverse engineering concerns Perfect static disassembly without metadata is provably undecidable Best disassemblers (IDA Pro) make many mistakes ProgramInstruction Count IDA Pro Errors mfc42.dll mplayerc.exe vmware.exe

Infeasibility of Perfect Disassembly Disassemble this hex sequence Undecidable problem FF E0 5B 5D C3 0F F 84 EC 8B Valid Disassembly FF E0jmp eax 5Bpop ebx 5Dpop ebp C3retn 0F F 84 EC jcc 8B …mov Valid Disassembly FF E0jmp eax 5Bpop ebx 5Dpop ebp C3retn 0Fdb (1) F 84 EC mov 8B …mov Valid Disassembly FF E0jmp eax 5Bpop ebx 5Dpop ebp C3retn 0F 88db (2) 52push edx 0F 84 EC 8B … jcc 9

Original Binary Reins Binary Separating Code from Data Header IAT.data.text Original Memory Layout Rewritten Header IAT.data.told (NX bit set) Rewritten Memory Layout.tnew (NW bit set) Denotes a section that is modified during static rewriting 10 High Memory Low Memory kernel32.dlluser32.dll kernel32.dll

De-Shingling Disassembly HexPath 1Path 2Path 3Path 4 FFjmp eax E0loopne 5Bpop 5DL1: pop C3retn 0Fjcc 88mov B0mov 50 FFN/A FF 8BL2: mov Byte Sequence: FF E0 5B 5D C3 0F 88 B0 50 FF FF 8B 11 Disassembled Invalid Included Disassembly jmp eax pop L1: pop retn jcc L2: mov loopne jmp L1 mov jmp L2

Aligning Instructions 12 Original Binary 0x68900Fmov eax, 0x6891D8 0x689015add eax, 1 0x68901Bcall eax …… 0x6891D9push ebx 0x6891DAmov ebx, [esp+4] Rewritten Binary 0x78900Fnop 0x789010mov eax, 0x6891d8 0x789016add eax, 1 0x78901Cnop (x4) 0x789020nop (x8) 0x789028and eax, 0x0FFFFFF0 0x78902Ecall eax 0x789030… 0x7892E0push ebx 0x7892E1mov ebx, [esp+4] 0x7892E5… Chunk instructions to 16 byte boundaries with targets at the beginning, and calls at the end [McCamant and Morrisett. USENIX 2006] Alignment nops Injected Instructions

Rewritten Binary Preserving Good Flows 13 Original Binary 0x68900Fmov eax, 0x6891D8 0x689015add eax, 1 0x68901Bcall eax …… 0x6891D9push ebx 0x6891DAmov ebx, [esp+4] Turn original code section into a dynamic lookup table.told0x6891D90xF4 loc_7892F0.tnew0x78900Fnop 0x789010mov eax, 0x6891d8 0x789016add eax, 1 0x78901Cnop (x4) 0x789020cmp 0xF4, [eax] 0x789023cmovz eax, [eax+1] 0x789027nop 0x789028and eax, 0x0FFFFFF0 0x78902Ecall eax 0x789030… 0x7892F0push ebx 0x7892F1mov ebx, [esp+4] 0x7892F5… Alignment nops Injected Instructions

Preserving Good Inter-module Flows 14 jmp [IAT:CreateWindow] Original CodeRewritten Code CreateWindow jmp [IAT:CreateWindow] CreateWindow IAT data section locked non-writable

Computed Inter-module Flows computed jumps to trusted modules dynamic linking (DLLs) callbacks (event-driven programming) 15 trusted library intermediary library (trusted) rewritten code caller callback stub callback_ret callback return trampoline

Results 16

IRM Synthesis Enforced policies on Eureka client (>1.6MB code): Disallow creation of.exe,.msi, or.bat files Disallow execution of Windows explorer as an external process Disallow opening more than 100 SMTP connections Malware policies: Disallow creation of.exe,.msi, or.bat files Successfully stopped virus propagation for real world malware samples 17 Policy-adherant binary Policy Rewriter Binary

TCB Formal Verification Formal verification of rewritten binaries 1500 SLOC of 80-column OCaml code no shared code between verifier and rewiter median verification time: 0.4 ms/KB code Allows rewriter to remain completely untrusted! rewriting deployable as an untrusted service 18 Policy-adherant binary Policy Rewriter Binary Verifier

Compatibility Limitations COM objects Runtime code generation (JIT) Undocumented OS callbacks 19

Conclusion Reins finally opens the door to full-scale COTS native SFI for massively complex, real-world applications without source. no source code, debug info, or disassembly (assumed unavailable) compiler-agnostic real COTS binary features interleaved code and data, computed control-flows, dynamic linking, event- driven callbacks, multithreading automated synthesis of monitor from policy specification automated machine-verification low runtime overhead (~2.4%) successfully tested on real commercial applications (>3MB code) Practical Applications: safe reuse of untrusted commercial software in security-critical environments rewriting on demand: rewriter deployable as an untrusted third-party service due to separate verifier 20

References R. Wahbe, S. Lucco, T. E. Anderson, and S. L. Graham. Efficient software-based fault isolation. In Proc. ACM Sym. Operating Systems Principles, pages 203–216, F. B. Schneider. Enforceable security policies. ACM Trans. Information and Systems Security, 3(1):30–50, M. Abadi, M. Budiu, U. Erlingsson, and J. Ligatti. Control-flow integrity. In ACM Conference on Computer and Communications Security, pages , S. McCamant and G. Morrisett. Evaluating SFI for a CISC architecture. In Proc. USENIX Security Sym., Ú. Erlingsson, M. Abadi, M. Vrable, M. Budiu, and G. C. Necula. XFI: Software guards for system address spaces. In Proc. Sym. Operating Systems Design and Implementation, pages 75–88, H. Shacham. The geometry of innocent flesh on the bone: Return-into-libc without function calls (on the x86). In Proc. ACM Conf. Computer and Communications Security, pages 552–561, B. Yee, D. Sehr, G. Dardyk, J. B. Chen, R. Muth, T. Ormandy, S. Okasaka, N. Narula, and N. Fullagar. Native Client: A sandbox for portable, untrusted x86 native code. In Proc. IEEE Sym. Security and Privacy, pages 79–93,

Advantage over VMs no air gap IRM has controlled but direct access to system resources and other processes no semantic gap no dynamic instruction interpretation or translation better performance fewer context switches light-weight VM logic essentially in-lined into code formal verification few VMs have been formally verified each change to VM (e.g., to enforce new policy) requires re- verification of VM 22