1. HookScout: Proactive Binary-Centric Hook Detection
Heng Yin, Pongsin Poosankam, Steve Hanna,
and Dawn Song
Problem statement

2. What is hook?
Malware registers its own function (i.e. hook) into the target location
Later, data in the hook site is loaded into EIP, and the execution is redirected into malware’s own function.
NewZwOpenKey
For those not familiar with the concept of hook, I will explain it with a concrete example.
Execution is redirected
Install the address of NewZwOpenKey
ZwOpenKey
SSDT (System Service Descriptor Table)
an example of SSDT hooking

3. Hooking is an important attack vector
malware often needs to install hooks to implement illicit functionalities
Rootkits want to intercept and tamper with critical system states
Network sniffers and stealth backdoors intercept network stack
Spyware, keyloggers and password thieves need to know when sensitive info arrives
Why is hook so important? This is because malware often needs to divert system’s execution flow to build illicit functionalities. As you can see rootkits, network sniffers, stealth backdoors, spyware, keyloggers, password thieves, they all install hooks to achieve some malicious intents.

4. Hooking Techniques Are Evolving
Old Technique: SSDT, IDT, IAT, EAT, etc.
Defeated by many existing hook detection tools
New trend: function pointers in kernel data structures
IO completion routines
APC queues
Threads saved context
Protocol Characteristics Structures
Driver Object callback pointers
Timers
DPC kernel objects
DPC scheduled from ISR
IP Filter driver hook
Exception handlers
Data buffer callback routines
TLS callback routines
Plug and play notifications
All kinds of WDM driver stuff
Many more, …
Old-fashioned malware installs hooks in some well-known memory regions, such as …
Many existing hook detection tools check these regions, and detect hooks easily.
To evade detection, malware starts to hook un-explored memory regions, more specifically, function pointers in kernel data structures. Researchers have created a long list for function pointers that can be potentially exploited. It is actually far from being complete.

5. Advantages of Function Pointer Hooking
Attack space is vast
~20,000 function pointers in Windows kernel
Hard to locate and validate
~7,000 in dynamically allocated memory regions
Many of them in polymorphic data structures
A polymorphic hash table in Windows kernel

6. Example: A polymorphic linked list
FILE_OBJ
DEVICE_OBJ
FILE_OBJ
head
head
head
ObjListHead
open
state
open
ioctl
This is an example of type polymorphism. It is a linked list. Nodes are linked with each other using a LIST_ENTRY, which essentially consists of two pointers. This is a polymorphic linked list, because nodes in it have different types, except that they share a common header structure.
Static source code analysis tries to determine the type of each structure statically, and generate a type graph, and then use this type graph to traverse the data structure.
So in this case, it is able to determine that this global variable is a linked list, but it cannot determine the type of node in this linked list, because there can be two different types of nodes in this linked list.
Polymorphic data structure is not uncommon. In Windows, there is a centralized hash table to manage many different important kernel objects. So dealing with polymorphic data structures is very important.

7. Our Goal
Given the binary distribution of an OS kernel, automatically generate a hook detection policy
Locate function pointers
Deal with polymorphic data structures
Validate function pointers
only 3% ever change in their lifetime
Simple policy: check if constant function pointers ever change

8. System Overview

9. Monitor Engine Goal: determine concrete memory layout Solution:
For each static/dynamic memory object, determine primitive types for each memory word
Primitive types: NULL, FP, CFP, DATA
Solution:
Monitor memory objects
Track function pointers
Addr=e h
Size = 20
We perform binary analysis on the OS kernel. We monitor OS execution in the binary level, and track function pointers and memory objects, and then we get concrete layout for each memory object. Then we infer the hook detection policy from these concrete memory layouts. We perform context-sensitive policy inference.
DATA
DATA
CFP
NULL FP

10. Monitor Engine: Monitor Memory Objects
Run the guest OS within TEMU
TEMU: a whole-system binary analysis platform, based on QEMU
For dynamic objects: Hook memory allocation/deallocation routines
ExAllocatePoolWithTag, ExFreePool
RtlAllocateHeap, RtlFreeHeap
For static objects: Hook module loading routine
MmLoadSystemImage
Addr=e h
Size = 20
We perform binary analysis on the OS kernel. We monitor OS execution in the binary level, and track function pointers and memory objects, and then we get concrete layout for each memory object. Then we infer the hook detection policy from these concrete memory layouts. We perform context-sensitive policy inference.

11. Monitor Engine: Track Function Pointers
CreateFile() {
FILE_OBJ *f = malloc(sizeof(FILE_OBJ)); …
f->open = MyFileCreate;
InsertListTail(&f->link, &ObjListHead); }
804d7200: call malloc …
804d7230: mov [ebp-50h], 805d5141h
Appear in relocation table
Point to a function entry
Addr=e
Size = 40
Caller=804d7200
Addr=e
Size = 40
Caller=804d7200
We monitor the execution of the OS kernel at binary level. Suppose the left side is the actual source code, and the right side is the instructions being executed.
When we see a memory object is allocated, we check the execution context, that is, we check who is the caller. That is we monitor all memory objects, and their caller information. Here we see a function pointer is assigned to a memory location. It is not so straightforward to observe this at binary level.
Then we know locations of all function pointers in the kernel space, and the relative locations in each memory object. Then we determine the type of each word in the memory object. We have four primitive types: DATA, FP, CFP, and NULL.
How can we tell this constant number is an initial function pointer? We have an important observation. A function pointer is an absolute address to a function entry in a relocatable kernel module.
DATA
DATA
DATA
CFP
NULL FP

12. Inference Engine Goal: Infer abstract memory layout
Approach: context-sensitive abstraction
Notion: Object creation context is the execution context where an object is created (e.g., caller of malloc)
Binary point of view: return addresses on the call stack
Rationale: Objects created under the same context have the same type
Solution: Merge concrete layouts with the same context into an abstract layout

13. Inference Engine: Context-Sensitive Type Inference
Addr=e
Size = 40
Caller=804d7200
Addr=e
Size = 40
Caller=804d7200
Generalized Layout
caller=804d7200
DATA
NULL
DATA
DATA
DATA
DATA
DATA
DATA
DATA + =
CFP
CFP
CFP
NULL
CFP
CFP FP
CFP FP
To infer a generalized layout, we merge the memory objects allocated under the same context into a single template

14. Detection Engine Goal: Solution: Implementation:
Enforce the hook detection policy on user’s machine
Solution:
Monitor memory objects
Hook the same set of functions
Apply the abstract layout
Use the return addresses as the key to the abstract layout
Implementation:
Kernel module vs. Hypervisor

15. Detection Engine: go back to the example
FILE_OBJ
FILE_OBJ
DEVICE_OBJ
head
head
head
ObjListHead
open
state
open
ioctl
Abstract Layout
(caller=804d7200)
Abstract Layout
(caller= )
DATA
DATA
DATA
DATA
DATA
DATA
DATA
DATA
CFP
DATA
CFP

16. Experimental Evaluation
Aspects to Evaluate
Attack Space
Analysis subsystem: policy coverage
Detection subsystem:
realworld rootkits/performance/false alarms
Experimental Setup
Host machine: 3.0GHz CPU 4 GB RAM Ubuntu
Guest machine: 512MB RAM Windows XP SP2

17. Evaluation: Attack Space

18. Evaluation: Function Pointer Lifetime Distribution

19. Evaluation: Policy Generation

20. Evaluation: Realworld Rootkit Detection

21. Evaluation: Performance of Detection Subsystem

22. Conclusion Function pointer hooking is a new trend
Large attack space
Hard to detect
Without OS source code, even harder
We developed HookScout
Binary-centric: deal with OS binary code
Context-sensitive: deal with type polymorphsim
Proactive: detect attacks in advance