Operating System Protection Through Program Evolution Fred Cohen Computers and Security 1992

Ultimate attack ● Physical access to device ● Disassembly and reverse engineering – “Any finite state machine can be examined and modified at will” – One can only delay this attack by increasing complexity of operation ● Widespread static defenses problematic – One only needs to craft a single attack to produce a widespread virus

Ultimate defense ● Drive complexity up to make the cost of reverse- engineering too high to be worth it – Security through obscurity ● Force attacker to use reverse-engineering to attack additional machines – Evolve defenses Current operating systems do not obscure nor do they evolve

Tools for obscuring and evolving ● Instruction equivalence – Replace equivalent instructions ● (e.g. mov eax,0 => xor eax,eax) – No impact on time or space, slight impact on a serious attacker

Tools for obscuring and evolving ● Equivalent instruction sequences – Replace instruction sequences with equivalent ones ● (e.g. add eax, 17 => add eax,20 + sub eax,3) – May increase time and space ● Space-equivalent replacements difficult – Can complicate things for attacker by obscuring control flow if x=0 goto A... A: goto x* :....+6: goto x :goto A.+2:... A:... y=(x+17)*35 if x <= (35*17) goto A... A:...

Tools for obscuring and evolving ● Instruction reordering – Permute into equivalent sequences – No additional time or space, but may not impact attacker a=a+1 b=b*3+c j=j+12 a=a+1 j=j+12 b=b*3+c a=a+1 j=j+12 etc....

Tools for obscuring and evolving ● Variable substitutions – For high-level languages, may not work unless compilers produce resorted symbol tables – At lower levels, alter locations of memory storage areas 1: 2705 ;jump to 5 2: 0 ;loc of 'A' 2: 1701 ;loc of 'B' 3: 1701 ;loc of 'B' 3: 2103 ;loc of 'C' 4: 2103 ;loc of 'C' 4: 0 ;loc of 'A' 5: 1002 ;A=A+1 5: 1004 ;A=A+1 6: 2003 ;B=B-1 6: 2002 ;B=B-1 7: 2103 ;shift B left 7: 2102 ;shift B left 8: 1734 ;xor C with B 8: 1723 ;xor C with B 9: 4306 ;B=0?goto 6 9: 4206 ;B=0?goto 6 10: 0 ;halt CPU

Tools for obscuring and evolving ● Adding and removing jumps – Increases time and space linearly with number of jumps – Easy to evolve in conjunction with instruction and instruction sequence reordering via “jump tables” A:I_1A: Jump A' B:I_2 B: Jump B' C:I_3 C: Jump C' D:I_4 D: Jump D' E:I_5 E: Jump E' F:... F:... B':I_2;Jump C E':I_5;Jump F C':I_3;Jump D A':I_1;Jump B D':I_4;Jump E

Tools for obscuring and evolving ● Adding and removing calls – Replace in-line sequences with subroutine calls (and vice versa) – Impacts stack and processor information – May increase time and space goto Start; goto Start; f(a): return(2*a); o(): z=z+3; pr(a): wait-for-printer(); return(0); print(a); Start: x=3; return(0); y=2*x; Start: x=3; o(); y=f(x); x=z+y/2; z=z+3; wait-for-printer(); x=z+y/2; print(x); pr(x); halt; halt;

Tools for obscuring and evolving ● Garbage insertion – No limit to equivalent programs that can be generated – Increases time and space – Fools attacks that look for specific instruction sequences ● Wheat vs. chaff Start: a=21; b=-19; for d=b to 98 step 7 do c=c+d; OScall(a,4,d) done where Oscall(x,y,z) invalid unless x=21,y=4,d=2

Tools for obscuring and evolving ● Program encodings – Produce random sequence of instructions – XOR with a set of randomly selected bits – Run-time decoding/decryption produces time/space overhead – Strong against attacks involving code inspection ● Must decode before inspecting ● Combined with program evolution => must decode for each attack ● Attacker could wait until run-time to analyze

Tools for obscuring and evolving ● Simulation – Replace instruction sequence with equivalent sequence for a different processor – Simulate other processor via interpretation – Processor being simulated can evolve – Very high time-space overhead

Tools for obscuring and evolving ● Build and execute – Build instructions prior to execution and then execute ● Self-modifying code ● Obscures instructions being executed until just before execution – Add randomness and evolution

Tools for obscuring and evolving ● Induced redundancy – Repeat test procedures used to make decisions – Evolve the type and number of tests randomly count=0;... x=open-file(`zz')... if (x <3) count++;... if (x >0) count++;... if (count 2) nogood;... x=open-file(`zz')... if (x = 2) OR (x = 1) count++;... if (count 3) nogood;... x=open-file(`zz')... if (x AND 3) > 0 count++;... if ((count AND 4) != 4) nogood;...

Tools for obscuring and evolving ● Intermixing programs – Interleave independent programs – Complex to analyze and implement (too many interactions amongst memory and register states) Subroutine 1 Subroutine 2 Mixed Subroutine s1(i,j):= s2(i,r):= sb(i,j,r):= x=0; x2=17; y=i+12; if (i <3) x=x+6; y=y*r/3.74; x=x*i+j/17; return; return; return;

Tools for obscuring and evolving ● Anti-debugger mutations – Disassemblers for systems with multiple instruction lengths ● Must be able to synchronize with instruction stream ● Pad stream with random garbage especially after jump instructions to confuse ● Place instructions in padding and jump into them ● Can only tell at run-time the instructions being used and their alignment

How to use the tools ● How mix techniques? – Based on performance desired ● When to evolve? – At the factory – After installation (bad for integrity checkers)

Experimental results ● Prototype built – Negligible performance impact – Limit evolution within certain areas of program – Can't quantify impact on attacker

Attacks on evolutionary defenses ● Points of attack – System calls are static – Must evolve OS ● Tracing attacks – Perfect simulation cannot be avoided – Instruction redundancy forces attacker to trace each attack repeatedly in order to bypass defenses systematically ● Size of space vs. complexity ● Computational advantage – Much like cryptosystems, obfuscation only changes computational advantage