VMM Based Rootkit Detection on Android Class Presentation Pete Bohman, Adam Kunk, Erik Shaw

Motivation The Increase of Mobile Malware Variants (2004 – 2010) Smartphone malware on the rise Increased security implications (compared to PC) Sensitive information: GPS, contacts, SMS, call log Constantly connected to the Internet Naïve users, limited use of anti-virus protection

Defensive Rootkit Approaches User mode rootkits – Process infection, binary patching, lib hooks User mode integrity checkers – tripwire, chkrootkit, rkthunter, AV scanner Kernel mode rootkits – malicious device drivers and LKMs – sys call hooking, kernel data structure manipulation (DKOM) Kernel level inspection – behavioral analysis, data structure integrity checkers, hook detection But… Any kernel level inspection mechanisms can be subverted by kernel level rootkits!

Our Approach Two Pronged KM security mechanisms System call integrity checks Hidden process detection ACLs for Android personal data VMM inspection Ensures integrity of static KM Isolated from host OS We exercise a “layer-below” level of security in which we establish trust beneath the kernel Android Software Stack

Overview Design VMM Interface Design Protection KM Design Implementation Results Demo Presentation Conclusion Q&A

VMM Interface Design Android VMM Hardware (Emulator) Linux Kernel Trusted KM Libraries and Runtime Application Framework 1. Hardware Timer Interrupt 2. Validate Protected KM3. Raise Monitor Interrupt 4. Invoke KM

Protection KM Design Linux Kernel Trusted KM System Call Whitelist Original Sys Call Table Libraries and Runtime Maps Application Framework ContactsSMSApp Content Provider Location Provider Activity Manager Open Malicious Native Application System Calls Open Socket Read GPS SQL Query Sys Call Table …

Protection KM Design Linux Kernel Trusted KM System Call Whitelist Original Sys Call Table Libraries and Runtime X Malicious LKM System Calls Sys Call Table Y Z Android VMM Monitor Interrupt Linux Kernel

Implementation VMM Interface Implemented VMM security functionality in an emulated hardware device within QEMU Protected KM data and text compiled into QEMU emulator (VMM) Linux Kernel Source with Protection KM Compilation Kernel Image Protected Text Sect. Protected Data Sect. QEMU Emulator (VMM) Compilation QEMU Emulator(VMM) Protected Text Protected Data

Implementation Protection KM Checks system service function pointers in sys_call_table Reports problem and restores them if they have been altered Hooks sys_open and prevents access to mmssms.db and contacts2.db Unless process name and parent process name are found in the access control list Likewise restricts socket and sys_read of GPS data according to the access control list

Implementation Malicious native mode application Reads contacts database file Reads GPS location Transmits data using network sockets Runs as Linux user mode process, underneath the Android Application Framework Malicious LKM Implemented by following the MindTrick Rootkit techniques Intercepts read system calls to access GPS location Attempts to access the SMS and contacts database files

Results We are able to detect and correct modifications to the sys_call_table Malicious LKM system call table hooking was thwarted Malicious LKM unable to obtain GPS coordinates We are able to prevent malicious access to sensitive resources Malicious application fails to read GPS data, fails to open sensitive database files, and fails to open network sockets Malicious LKM also fails to open sensitive database files Demo (short): Demo (long):

Conclusion “Layer-Below” Protection Security of the Linux kernel must be rooted in a layer below the kernel to withstand kernel level attacks Trusted “layer-below” protection mechanisms can serve as the secure foundation for enabling additional advanced protection mechanisms in the kernel

Questions?

Backup – Android Architecture

Backup – Protection by Layer App Permissions Dalvik VM Isolation App signatures Limits application abilities in order to prevent malicious behavior. Virus Scanners Remote Lockout Modified system binaries Trojan’d services Stolen device Linux user and group permissions Access control Architecture LayerSecurity MechanismThreat Mitigation

Backup – Problem Statement Rootkit detection and prevention on the Android platform with specific regards to the sensitive resources Android provides. Kapersky 2011: 1046 unique malware strains targeting mobile platforms Android platform built on Linux Kernel, a well known target. Sensitive information on smart phones GPS, contacts, text messages, call log