A Binary Agent Technology for COTS Software Integrity Richard Schooler Anant Agarwal InCert Software

The Mission Critical Environment Output Input Operating System SAP COTS Binary The development environment The deployment environment COTS Binary

Objective COTS Binary Input Output Operating System SAP To improve the integrity of the deployment environment with COTS software in the presence of attacks, bugs

Our Focus COTS Binary Input Output Operating System SAP Outer security defences will be breached by attackers Use a practical, systems level approach – execution- time monitoring On COTS program or data corruption, rapidly d- detect problems a- trigger an alarm p- try to protect r- recover

Our Approach: Execution-Time Monitoring of COTS through Binary Instrumentation The development environment The deployment environment COTS New Missing source Legacy COTS Binary COTS New Missing source Legacy d- Policy specs for detection d- Heartbeat insertion d- Argument range checks d- Rare code execution/sigs. a- Alarm messages to console p- Defaults for fault tolerance p- Access constraints, redund. r- Logging COTS New Missing source Legacy

Our Approach to Integrity l A pragmatic system level approach l Works with COTS binaries, even legacy codes l Allows a user to establish desired security levels and to some extent modify policy on the fly l Allows monitoring of all user code execution, not selected OS calls l Can work completely at the user’s deployment site

Drawbacks of Binary Insertion l Specific to a single platform, needs new technology development for different platform l Challenging to relate low-level observable events back to high-level user actions n hard to detect some types of intrusions that only affect data corruption n hard to protect or correct problems at higher semantic levels

Three Major Components in the Prototype, Three Major Tasks l Core technology for customizable agent insertion into PC/NT, PC/Linux l Anomaly detection and reporting l Rapid recovery and problem pinpointing

Selected Challenges and Ideas l Core technology for agent insertion into binary n How to deal with unknown relocations, e.g., for dusty decks – incremental control and dataflow analysis; an integrated static and dynamic method n Dealing with multithreading, in particular, time syncing and monitoring events in a distributed environment l Anomaly detection n Runtime comparison against execution path signatures? n State machines for control flow checks (e.g., Abraham) n How to minimize runtime overhead – borrow compiler optimization techniques (e.g., steal registers, inline code, sampling, multilevel checks) l Rapid recovery and problem pinpointing technology n Instrumentation records trace in circular buffer to pinpoint problem and facilitate recovery from crash/alert following an actual or suspected attack n Can we get data values? Use dataflow analysis and offline simulation to obtain intermediate data values

Measures of Success l Core technology for agent insertion into binary: n Can we handle all binaries, even dusty decks? n Target: Performance degradation to be under 1 percent l Anomaly detection n What fraction of injected problems can we detect u automatically u with user spec l Rapid recovery technology n Performance degradation to be under 1 percent n Can we cut recovery time significantly? We will measure recovery time with and without n As a bonus, can we catch problems before system goes down? l We are building a prototype system, work with real users, and measure

Current Progress l Work on NT binary insertion prototype ongoing l Demo of early capability showing n instrumentation n simple recovery log n detecting application has crashed n taking control and n writing out log l Handling multithreading, DLLs imminent l Ongoing thinking on detection capability

AGENT test al,0x3 jnz 0x1143 AGENT add ebx,ecx jc 0x1101 AGENT shr edx,0x1 add ebx,edx AGENT test al,0x3 jnz 0x1143 AGENT inc eax add ecx,edi add edx,esi cmp eax,0xa while ((c = ++ci)) { INSTRUCTION_ITERATOR ii = c->Instructions(); while ((inst = ++ii)) inst->Lift(null_state); while ((inst = ++ii)) inst->Lift(null_state); while ((c = ++ci)) { INSTRUCTION_ITERATOR ii = c->Instructions(); while ((inst = ++ii)) inst->Lift(null_state); while ((inst = ++ii)) inst->Lift(null_state); Instrumentation for Recovery: Basic Idea l At each program block, record the progress of program execution.

Instrumentation Runtime Naïve implementation: Write block id Increment index (in memory) Check for wrap-around Too big/slow: many instructions per block; large buffer DAG Header setup new trace record DAG Block 1 set bit in current record DAG id

Instrumentation Example 1 test al,0x3 jnz 0x1143 add ebx,ecx jmp short 0x1156 shr edx,0x1 add ebx,edx inc eax add ecx,edi add edx,esi cmp eax,byte 0xa jc 0x1101 Original

Instrumentation Example 2 mov eax,[0x300c] mov ecx,[byte eax+0xc] mov [byte eax+0xc],dword 0x1 mov edx,[byte eax+0x8] mov [dword eax+edx+0xffe],ecx add edx,0x4 and edx,[byte eax+0x4] mov [byte eax+0x8],edx test al,0x3 jnz 0x1143 or [byte ebp+0xe],byte 0x2 add ebx,ecx jmp short 0x1156 or [byte ebp+0xe],byte 0x1 shr edx,0x1 add ebx,edx or [byte ebp+0xe],byte 0x4 inc eax add ecx,edi add edx,esi cmp eax,byte 0xa jc 0x1101 Instrumentation Original

Instrumentation Example 3 mov eax,[0x300c] mov ecx,[byte eax+0xc] mov [byte eax+0xc],dword 0x1 mov edx,[byte eax+0x8] mov [dword eax+edx+0xffe],ecx add edx,0x4 and edx,[byte eax+0x4] mov [byte eax+0x8],edx test al,0x3 jnz 0x1147 or [byte ebp+0xe],byte 0x2 add ebx,ecx jmp short 0x1164 or [byte ebp+0xe],byte 0x1 shr edx,0x1 add ebx,edx or [byte ecp+0xe],byte 0x4 inc eax add ecx,edi add edx,esi cmp eax,byte 0xa jc 0x1103 Instrumentation Displacement Fixup Original

Instrumentation Example 4 mov eax,[0x300c] mov ecx,[byte eax+0xc] mov [byte eax+0xc],dword 0x1 mov edx,[byte eax+0x8] mov [dword eax+edx+0xffe],ecx add edx,0x4 and edx,[byte eax+0x4] mov [byte eax+0x8],edx test al,0x3 jnz 0x1147 or [byte ebp+0xe],byte 0x2 add ebx,ecx jmp short 0x1164 or [byte ebp+0xe],byte 0x1 shr edx,0x1 add ebx,edx mov ecp,ebp or [byte ecp+0xe],byte 0x4 inc eax add ecx,edi add edx,esi cmp eax,byte 0xa jc 0x1103 Instrumentation Displacement Fixup Original Register Transfer

Related Approaches l Applied at source level during development n e.g., type based safety; work of Lee et al. l Applied at link time with special object formats n e.g., software fault isolation; work of Pandey et al. l Applied through interpretor n e.g., safe Java interpreters l Applied during program execution – works with existing COTS n e.g., sandboxing through middleware software n e.g., modify OS with extensions to catch OS calls, Ghosh n e.g., wrap application to steer OS calls to intermediary, Ghosh

The Current Commercial Solution: Middleware COTS Binary Input Output Operating System SAP Middleware wrappers Slow Cannot deal with viruses Cannot improve fault tolerance of COTS package itself

Sandboxing through OS Call Interception (OS extensions or wrapping executable) COTS Binary Input Output Operating System SAP This is the right first step, and less intrusive than binary instrumentation Cannot improve fault tolerance of COTS package itself Cannot handle communication through shared memory Cannot observe internal app behavior OS extensions can slow OS for others

Summary l A systems approach to COTS Integrity l Approach based on execution-time monitoring using binary insertion l We have an early prototype version of NT binary insertion implemented