Operating System Protection Through Program Evolution By Dr. Frederick B. Cohen Presented by William Lu

The Ultimate Attack How to defeat defensive measures of a system? Gain physical access to the system Reverse engineer defenses Find weak link and exploit

The Ultimate Defense How to defend against attackers Make attacks extremely complex Make costs too high to be worth attacking i.e. passwords Large space Spread out probability density (diffusion) Obscuring stored information (confusion) How to defend against attackers In practice, password space is huge but human methods of password selection leads to a high probability of select words.

The Ultimate Defense Current operating systems Space is enormous (all programs that fit in memory) High probability subspace (very small number of versions) No confusion No confusion – the part of the program that performs any given operation is apparent

The Ultimate Defense How to increase operating system defenses? Reducing coherence How? Unique defense for each system Feasibility? Too many unique defenses to design Compromise? Implement a fixed number of defenses

The Ultimate Defense More practical solution? Goal? Evolutionary defenses Goal? Produce a large search space Provide confusion Provide diffusion With more evolution, there’s less performance, but higher cost of attack

(some) Techniques for Program Evolution Equivalent instruction sequences Instruction reordering Variable substitution

Equivalent Instruction Sequences What does it do? Replaces instruction sequences with equivalent sequences i.e. add 17 is equivalent to add 20 and subtract 3 How does it help defend against attacks? As evolution increases so does time and space Potentially infinite evolutions Creates enormous possible executions

Instruction Reordering What does it do? Reorders instructions without altering program execution Order does not matter How does it help defend against attacks? Typical system call involves setting a series of values. Values are independent and can be set in any order. Assigning independent values can be reordered Increases complexity of attacks to n! different orderings (n = # of different instructions)

Instruction Reordering 3 different instructions 6 different forms … I=3 J=5 K=8

Variable Substitution What does it do? Alters the location of memory storage areas How does it help defend against attacks? Prevents static examination and analysis of parameters Move variables to different locations to prevent patterns without affecting program execution

(and other) techniques? What to do with these (and other) techniques?

Providing Evolution in Defenses How to evolve? Select a mix of evolution techniques Increase complexity while minimizing impact on end users

Providing Evolution in Defenses When to evolve? At the factory? Uniquely identify each disk sent out Reduced efficiency At installation? Unique tracking number, unique registration numbers Evolution at installation can take place while installer is asking for user input or swapping of disks. Crucial to have unique and confidential evolution Takes time at end user level

Providing Evolution in Defenses When to evolve? After installation? Cannot assure against corruption Cannot trust internal checking Attack that succeeds on one day may fail the next Unique tracking number, unique registration numbers Evolution at installation can take place while installer is asking for user input or swapping of disks.

Attacks on Program Evolution Points of Attack Tracing Attack

Point of Attack How? How to defend against it? Find original entry point and exploit it Gain direct access to hardware or operating system internals How to defend against it? Evolve the core of the operating system Evolving the calling mechanism Calls that bypass protection may be of the wrong form Used by viruses

Tracing Attacks How? How to defend against it? Trace programs at execution or simulation How to defend against it? Use redundancy Force attacker to use tracing on each attack

Conclusion Program evolution can increase the complexity for an attacker Create a large search space to make attacks infeasible Need more study to reach maturity