A Binary Agent Technology for COTS Software Integrity Anant Agarwal Richard Schooler InCert Software

The Mission Critical Environment Output Input Operating System SAP COTS Binary The development environment The deployment environment COTS Binary

Objective COTS Binary Input Output Operating System SAP To improve the integrity of the deployment environment with COTS software in the presence of attacks, bugs

Assumptions and Scope COTS Binary Input Output Operating System SAP Outer security defenses will be breached by attackers Use a practical, systems level approach – execution- time monitoring On COTS program or data corruption, rapidly d- detect problems a- trigger an alarm p- try to protect r- recover

Our Approach: Execution-Time Monitoring of COTS through Binary Instrumentation The development environment The deployment environment COTS New Missing source Legacy COTS Binary COTS New Missing source Legacy d- Policy specs for detection d- Heartbeat insertion d- Argument range checks d- Rare code execution/sigs. a- Alarm messages to console p- Defaults for fault tolerance p- Access constraints, redund. r- Logging COTS New Missing source Legacy

Drawbacks of Binary Insertion l Specific to a single platform, needs new technology development for different platform l Challenging to relate low-level observable events back to high-level user actions n hard to detect some types of intrusions that only affect data corruption n hard to protect or correct problems at higher semantic levels

Three Major Components in the Prototype, Three Major Tasks l Core technology for customizable agent insertion into PC/NT l Anomaly detection and reporting l Rapid recovery and problem pinpointing

Selected Risks/Challenges and Mitigation l Core technology for agent insertion into binary n Dealing with real environments – e.g., multithreading and synchronization, in particular, time syncing and monitoring events in a distributed environment n How to minimize runtime overhead – borrow compiler optimization techniques (e.g., steal registers, in ine code, sampling, multilevel checks) n How to deal with unknown relocations, e.g., for dusty decks – incremental control and dataflow analysis; an integrated static and dynamic method l Anomaly detection – can we catch problems without user help? n Runtime comparison against execution path signatures? n State machines for control flow checks (e.g., Abraham) l Rapid recovery and problem pinpointing technology n Third party problems n Can we get data values? Use dataflow analysis and offline simulation to obtain intermediate data values

Measures of Success l Core technology for agent insertion into binary: n Can we handle all binaries, DLLs, even dusty decks? n Target: Performance degradation to be under 1 percent l Anomaly detection n What fraction of injected problems can we detect l Rapid recovery technology n Can we cut recovery time significantly? We will measure recovery time with and without n As a bonus, can we catch problems before system goes down? l Build a prototype system, work with real users, and measure

Realistic Environments Have Multiple Threads and Modules DLL1 DLL2 T1 T2 T3

Multiple Threads – Per-DLL Buffer DLL1 Buffer DLL1 T1 T2 T3 DLL2 DLL2 Buffer TS Lock overhead Contention in SMPs TS Thread IDs

Multiple Threads – Per Thread Buffer Timestamps Sequence counter Ids DLL1 DLL2 T1 T2 T3 s4 s1 s2 s3 s s5

Multiple Machines? DLL T1, Mx T2, My T3, Mz s1 s2 s3 s4 s How to synchronize efficiently times at a fine grain? How to maintain efficiently a cross-machine counter?

Current Progress l Work on NT binary insertion prototype ongoing l Demo of early capability showing n instrumentation n simple recovery log n detecting application has crashed n taking control and n writing out log n user-requested snap-trace for hung or “molasses” mode n information viewer for multithreaded traces n some optimization l Handling multithreading, DLLs imminent – prototyped n needed significant changes to runtime system – leverage shared memory n ongoing thinking on distributed programs l Ongoing thinking on detection capability

Summary l A systems approach to COTS Integrity l Approach based on execution-time monitoring using binary insertion l We have an early prototype version of NT binary insertion implemented l We have also successfully instrumented multithreaded programs