1. Security Flaws 2 Ian Kayne
For School of Computer Science, University of Birmingham
13th November 2007

2. Approach Why are the principles important?
Expect unique systems & software
No courses on “Widgets v1.0 security”
Expect unusual problems
Expect unusual solutions
Expect issues outside your comfort zone
2/28/2019

3. Approach Your mission, should you choose to accept it…
95% of the time it’s (relatively) easy
Most attackers go for easy score
Losing sleep over “script kiddies”? Something’s gone wrong.
The other 5% is hard
It’s also the 5% that matters
Jack of all trades and master of some
Learn the principles, investigate the rest
2/28/2019

4. Recap Buffer Overflows Executable stack Unchecked buffer input
Shellcode into buffer
Overwrite return address
2/28/2019

5. Buffer Overflow Jump using register
Any register can point at the start of the buffer
Inject shellcode to buffer as normal
Overwrite return address with address of instructions that jump using the register
Near-guaranteed success!
data data s h e l l c o d e
return address
test ecx, ecx jmp eax
push
EAX
2/28/2019

6. Prevention Non-executable stack (“NX bit”) Address randomisation
Canary bytes
data data
return address
2/28/2019

7. Review Next topic is complex! Builds on previous principles Questions?
Comments?
Review Items?
2/28/2019

8. RCE Reverse (Code) Engineering – “reversing” What is it?
Why is it done?
Malware research & defence
System interoperability requirements
Review and audit of software/security system
Why is it useful to security specialists?
“Learn the principles”
2/28/2019

9. RCE Required knowledge/skills (x86)
Platform knowledge – stack, registers etc
“Some” assembly language
C/C++ & as many other languages as possible
Operating system (Windows) mechanisms
win32api
Toolset (debugger, disasm, hex editor…)
Mindset (patterns, logic)
2/28/2019

10. RCE Imagine a strong protection mechanism Username License key
Complex validation system
Crippled shareware-style functionality
2/28/2019

11. RCE
Reversing demonstration
2/28/2019

12. RCE After the demonstration, recap: 1 byte patch Analysed executable
Set breakpoints on likely API calls
Traced up the call stack
Analysed the code
Found the good boy/bad boy “switch”
Patched the jump “live” to test
Converted RVA to file offset, patched file
1 byte patch
2/28/2019

13. Protection Imagine a strong protection mechanism again
License key system
CRC
Anti-debugging techniques
Encryption
2/28/2019

14. Protection Encryption for software protection Symmetric encryption
Asymmetric encryption
Fancy a wager?
2/28/2019

15. Protection Encryption for protection
Data must be decrypted before use
Code must be decrypted before execution
UPX (packer), Armadillo, Themida…
Can be made very hard, but not impossible
Remember the jump loop – EB FE
Generics – break one, break all
Homebrew is risky – “learn the principles”
2/28/2019

16. Protection Some obfuscation techniques:
Encode obvious “beacon” strings
Avoid win32api/library functions:
bpx MessageboxA
Use alternative functions/mechanisms
E.g.: SetWindowPos instead of ShowWindow
Roll your own api/GUI functions
Can’t break on GetWindowText if you don’t use it!
Hide code within the executable
Self modifying code, PE sections etc
2/28/2019

17. Protection Some anti-debugger techniques
Deliberate exceptions (code in SEH)
Self-debugging (can’t “stack” debuggers)
Timers and counters
Alter DR0 – DR7 hardware debug registers
IsDebuggerPresent()
Check for/attack known debugger processes, windows, services, drivers… (Starforce)
2/28/2019

18. RCE Why are these low-level technical techniques important?
“Learn the principles”
Your first job: consultant to betting company about to release online gambling game
The basics:
Internet security
Server security
Data security
But… what about the end-user software?
2/28/2019

19. Opinions Morality, legality, viability of protection
Invasive protection: Starforce, SecuROM
SonyBMG – Celine Dion, Neil Diamond…
Anti-piracy measures protect content creators’ rights and revenues.
–or–
Anti-piracy measures are ineffective, alienate customers & create legal issues.
2/28/2019

20. Review Thank you! Questions Comments Items to review Further study
2/28/2019