1. Rootkits – Avoiding detection Tillmann Werner, werner@bonn.edu Seminar Computer Security, B-IT 2006-11-27

2. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 2 Agenda Motivation and definition, short history of rootkits Entering the kernel: Hiding and starting Operating system internals: How things get executed Modern rootkit techniques Covert channels for stealth communication Countermeasures against rootkits

3. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 3 Motivation Hackers want to keep access to a successfully compromized box. At the same time, they want to remain undetected and thus need to hide their presence and traces. Using conventional ways to remotely access a hacked box is often much too noisy. Once a system is under control, an intruder normally wants to install his own invisible backdoor. All hacker activities and data related to those activities shall be invisible to legitimate users. Any permanent trace should be avoided, if possible.

4. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 4 Definition „A rootkit is a set of programs and code that allows a permanent or consistent, undetectable presence on a computer.“ Source: G.Hoglund, J. Butler: „Rootkits“, ISBN 0-321-29431-9 „A rootkit is a set of software tools intended to conceal running processes, files or system data from the operating system.“ Source: Wikipedia Encyclopedia, http://en.wikipedia.org/wiki/Rootkit

5. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 5 Privileged access and stealth In a nutshell: It‘s all about permanent access and stealth. Once a hacker has administrative privileges, assuring permanent access is only a matter of her creativity. This presentation focusses on how to stay stealthy. Filtering I/O between two layers could conceal the presence of a rootkit. Operating System Hardware User Program

6. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 6 From early rootkits… First rootkits came up in the late eighties. They were written for UNIX-like operating systems (Windows was not yet around). Early versions consisted of a bunch of modified programs that replaced the original instances on a compromized box. Typical candidates are programs that are used to examine the current system status, like ls, ps, who, netstat, etc. Also, the login program was often modified to accept login attempts for a specially crafted user. Invoked by telnet, it enabled attackers to come back at any time they wanted.

7. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 7 … to modern rootkits The early rootkits were easily detectable: modified binaries differ in size or cryptographic hashes from the original versions. There are 1001 ways to explore the system status – one could even write own programs. Comparing their results reveals the presence of rootkit binaries. Modern rootkits step into a lower layer, the kernel. If they run in kernel mode, any userland program is under their control as well. User input to a program and its output can then already be filtered on the kernel level. Patching binaries is not necessary at all.

8. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 8 A rootkit design example Usermode Program Kernel Driver Keyboard Sniffer Packet Sniffer Main OS Kernel Modifications Stealth Protection User SpaceKernel Space TCP port for remote control

9. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 9 Entering the kernel: Hiding rootkit resources A common approach for hiding a rootkit is prefix-based filtering: A resource whose name matches a given prefix is considered to belong to the rootkit and is lurked to other programs. Such a prefix can be prepended to file names, directory names or program names (which are inherited by corresponding processes). A similar method allows for hiding network traffic from other processes. Relevant packets are equipped with a magic value in their payload. This simple technique enables an attacker to hide most of his activities quite successful.

10. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 10 Entering the kernel: Starting a rootkit A common technique is to implement the rootkit as kernel module or driver that can be loaded during runtime. Modification of a present driver is also possible. The kernel code itself can be altered to start a rootkit. Both the on-disk image and the running kernel must be changed (e.g. by altering /dev/kmem on Linux). A rootkit program can also get hooked into the system to load automatically during operating system startup, e.g. by patching /sbin/init or by using.ini files. The boot loader can be modified to apply patches to the kernel just before the start phase.

11. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 11 i386 architecture The i386 architecture provides four different protection domains in hierarchical order, so called „rings“. This allows for enforcing a security model on the hardware layer. Most operating systems only use the rings 0 and 3 for compatibility reasons. Ring 0 is also known als „protected mode“ oder „kernel mode“. On modern operating systems only the kernel is allowed to enter it. Ring 3 is generally called „user mode“ and is used to execute code in an unprivileged level, i.e. with memory protection. 0 1 2 3

12. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 12 Entering ring 0 So, how can a program execute privileged instructions? The operating system provides system calls to a userland process. Common examples are functions like read(), write(), or open(). System calls are not used directly. Instead, they are encapsulated in libraries that can be used in high level languages. Examples are the libc or glibc on UNIX-like systems. Some platforms, like Microsoft Windows, do not want a program to use the low-level libraries directly. They provide API functions that simplify common programming tasks, like controlling a GUI.

13. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 13 Program User Space Kernel Space System Call TableInterrupt Descriptor Table : flow of execution : memory pointer Import Address Table Operating system internals – how things get executed

14. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 14 Program User Space Kernel Space System Call TableInterrupt Descriptor Table : flow of execution : memory pointer Import Address Table API Function Library Function Operating system internals – how things get executed

15. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 15 Program Choose IH from IDT User Space Kernel Space System Call TableInterrupt Descriptor Table : flow of execution : memory pointer Import Address Table API Function Library Function Operating system internals – how things get executed

16. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 16 Program Choose IH from IDT Choose Syscall from SCT User Space Kernel Space System Call TableInterrupt Descriptor Table : flow of execution : memory pointer Import Address Table API Function Library Function Operating system internals – how things get executed

17. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 17 Program Choose IH from IDT Choose Syscall from SCT System Call User Space Kernel Space System Call TableInterrupt Descriptor Table : flow of execution : memory pointer Import Address Table API Function Library Function Operating system internals – how things get executed

18. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 18 Operating system internals – how things get executed Program Choose IH from IDT Choose Syscall from SCT System Call User Space Kernel Space System Call TableInterrupt Descriptor Table : flow of execution : memory pointer Return Import Address Table API Function Library Function

19. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 19 Possible locations for intervention Program Choose IH from IDT Choose Syscall from SCT System Call User Space Kernel Space System Call TableInterrupt Descriptor Table Return API Function Library Function Import Address Table 1 5 3 4 2

20. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 20 On Windows platforms, the import address table (IAT) of a program contains pointers to API or library functions. A rootkit can modify the table to make a pointer reference a modified function. This function at first execute hostile instructions and then eventually calls the original API function if revelation is not feared. But: Every application has ist own import address table and the rootkit has to place a DLL with its wrapper functions in the filesystem. IAT Hooking Program API Function Import Address Table Rootkit Function 1

21. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 21 Specially crafted system calls affect all calling programs. The system call could check given arguments in order to decide how to operate. If a detection of the rootkit is feared, an alternative routine is executed that returns a different, harmless value. Modification of system call code requires runtime kernel patching on multiple locations. This is dangerous as one single error could crash the operating system. Modifying System Calls 2 Return ? Patched Syscall

22. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 22 Altering the system call table is the most common rootkit technique. Some system calls are replaced with modified versions by changing the corresponding pointers in the syscall table. A rootkit system call can behave different depending on the calling process, e.g. an open() fails if the argument is a rootkit resource. Problem: System call tables are backed up multiple times. Integrity checks could easily reveal a rootkit‘s presence. System Call Table Hooking Choose Syscall from SCT System Call 3 System Call Table ? System Call Rootkit Function

23. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 23 Altering the System Call Selector 4 Choose Syscall from SCT System Call TableRootkit System Call Table Instead of altering the syscall table, use your own! The kernel must know where to find the rootkit‘s syscall table. This can easily be done by changing the code of the system call chooser routine such that it consults the rootkit‘s table instead of the original one. But again, altering kernel functions during runtime would be necessary.

24. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 24 The pointer to the syscall selector code in the IDT can be changed to point to a modified instance. This function basically behaves like the original syscall selector but uses a modified instance of the system call table which references certain hostile syscalls. Modification of the original system call selector is not necessary. In theory, one could even go further and use an alternative IDT. The address in the Interrupt Descriptor Table Register (IDTR) must then be set to the location of the rootkit table. Interrupt Descriptor Table Hooking Choose IH from IDT Syscall Selector Rootkit Syscall Selector Interrupt Descriptor Table 5

25. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 25 … and without execution flow modification? Hiding processes is also possible: Unlinking a rootkit from lists used by the kernel to manage active processes can fool tools like ps or top. Files can be camouflaged by placing data in the file system‘s slack space. While this is not totally safe, studies show that chances are good for such data to survive a very long time. 6 File System Rootkit Process

26. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 26 Stealth communication: Covert channels (1) A rootkit is only of value if it allows remote access. To hide remote logins, covert channels are used. This method abuses another legitimate communication channel in order to transmit commands and their responses by manipulating certain properties. Timing channels use the relative timing of events to code information. Storage channels code information into existing data. The information transmitted via a covert channel can also be cryptographically encoded or obfuscated using steganography.

27. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 27 Information can be stored in the identification field of IP packets. DNS queries can also contain information that can be interpreted differently. The protocol does not even require a direct connection. Timing channels can be formed by representing bits by the duration of a session, the interval between two arriving packets, … Information theory teaches that the possibility of covert channels cannot be completely eliminated. However, a statistical analysis leads to detection in many cases. Stealth communication: Covert channels (2)

28. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 28 Countermeasures against Rootkits Mandatory access control can be implemented to reduce the danger of a system level compromize. Anti rootkits can control kernel data and the execution flow process themselves. Regular checks could detect modifications. Host-based intrusion detection and prevention systems can check system integrity by maintaining a list of cryptographic hashes for important resources. Integrity checks are only trusted when executed offline, that means from a trusted media while the suspect system is not running.

29. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 29 The Future of Rootkits Operating systems like Windows Vista and up-to-date Linux versions make use ring -1 on modern hardware to implement virtualization. A recent area of research is the development of so-called virtual machine based rootkits. The projects Blue Pill and SubVirt published working proof-of-concept methods that move a running operating system into a virtual environment. This environment is controlled from the outside. Hence, it is impossible to detect the rootkit from within the compromized operating system instance.

30. © Tillmann Werner, University of Bonn Rootkits - Staying Stealth. Silde No. 30 References Greg Hoglund, James Butler, Rootkits – Subverting the Windows Kernel Addison-Wesley, 2006, ISBN 0-321-29431-9 Andreas Bunten, Rootkits – Techniken und Abwehr, Proceedings of 10. DFN-CERT/PCA-Workshop, 2003, ISBN 3-8330-0097-X Andrew S. Tanenbaum, Modern Operating Systems, Second Edition,Prentice Hall, 2001, ISBN 0-13-031358-0 Daniel P. Bovet, Marco Cesati, Understanding the Linux Kernel, O'Reilly, 2002, ISBN 0-596-00213-0 Greg Hoglund, A *REAL* NT Rootkit, patching the NT Kernel, Phrack 55-05, 1999 Black Tie Affair, Hiding out under Unix, Phrack 25-06, 1989 Halflife, Abuse of the Linux Kernel for Fun and Profit, Phrack 50-05, 1997 Invisible Things, http://invisiblethings.org Rootkit – Share your old stuff, keep your good stuff, http://rootkit.com US DoD, Covert Channel Analysis of Trusted Systems (Light Pink Book), Rainbow Series, 1993 Craig H. Rowland, Covert Channels in the TCP/IP protocol suite, 1997, First Monday 05/97