1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class Root Kit Protection and Detection Shane Canon October

1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class Outline ● Protection – St Michael – What it does – How it works – Strengths and weaknesses ● Detection – Statistical detection of modified system calls – Strengths and weaknesses

1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class St Michael ● Source Forge Project – – Extension of the St Jude Project ● Attempts to protect kernel from root kitting through various means – MD5 several critical memory regions – Makes backup of kernel text – Timers run periodic checks – Checks automatically run during module insertion

1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class Monitoring Critical Regions ● MD5 are calculated for several critical, non-volatile regions – Kernel text – System call table ● Timers run periodic calculations of the current MD5's and compare them to the saved values ● Several calls automatically trigger a check – Exit – Module create, delete, and init

1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class Recovering from attempts ● Backups are made of kernel text ● Backup is encrypted through simple means ● If changes are detected, the module will attempt to decrypt the backup and restore it. ● If too many attempts are detected, the system automatically reboots

1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class Other Measures ● Module goes to some lengths to conceal itself – Decrypts several backups – During module initialization, some data structures and text are erased – The module is “cloaked” by removing it from the linked list of loaded modules ● Implements checks of some file system data to detect changes (possibly by writing to the raw device). ● Implements immutable files to prevent modification of files like init, kernels, insmod, etc. ● Protects kmem

1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class StMichael – Strengths and Weaknesses ● Strengths – Significantly raises the bar to trying to insert malicious code into the kernel. – Protection of kmem makes it more difficult to insert malicious code – Hashes detect typical changes performed by a root kit – Any failed attempt will trigger warnings and log messages ● Weakness – If attacker knows StMichael is loaded, they can use a kernel module to try and determine locations of key SM data structures and disable it. – As new root kit methods are designed, St Michael has to be updated to protect against those.

1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class Root Kit Detection ● Phrack Article – Volume 11, Issue 59, Article 10 - Execution path analysis: finding kernel based rootkits by Jan K. Rutkowski ● Describes method for detecting root kits based on number of operations required to perform standard systems calls ● Proof of concept – PatchFinder implements the methods JR describes

1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class Root Kit Detection – Intro ● Current Kernel based root kits typically use modified system calls to hide malicious files, network connections, and processes. ● By their very nature these modified system calls perform various checks before calling the unmodified system call. ● Although these checks are usually simple, they do add additional instructions. ● If these additional instructions can be statistically distinguished, this can be used to detect a modified kernel.

1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class Detection – Instruction Counting ● If the processor allows, the instructions counted for performing various system calls can be counted. ● Intel ia32 supports a single step mode. However, the trace flag is automatically cleared when the processor receives an interrupt or exception. ● PatchFinder uses ptrace flag in processor descriptor to determine if the CPU trace flag should be enabled in kernel mode. ● Modifies system call entry point to test ptrace flag. ● Modifies sys_ptrace to retrieve the number of instructions.

1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class Detection – Testing ● Once the module is loaded, a base line can be created for a given system and kernel. ● Testing done with some published kernel root kits (adore, SucKIT, etc) are detected one way or another – Most root kits will show instruction counts that are significantly different than the baseline. – SucKIT is detected because no results are returned since the modified sys_ptrace is never called.

1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class Detection – Strengths and Weaknesses ● Strengths – Once again it makes it much harder to implement a rock solid root kit ● Weaknesses – The malicious code is already loaded. If patch finder (or a similar method) is expected, the malicious code can attempt to return the expected values. This might involve testing the trace flag. However, this would be pretty tricky and tedious and any mistakes would trigger an alarm.

1 Future Technologies Group Shane Canon, canon at nersc dot govSummer Linux Kernel Class Conclusions ● StMichael and PatchFinder can significantly raise the bar for implementing a fool proof kernel root kit. ● However, root kits are moving targets, so a static approach is unlikely to work forever.