1. HyperSpector: Virtual Distributed Monitoring Environments for Secure Intrusion Detection Kenichi Kourai Shigeru Chiba Tokyo Institute of Technology

2. Distributed intrusion detection system (DIDS)  Useful to achieve self-monitoring of distributed systems ◆ Towards self-protection  Consists of multiple IDSes ◆ Including Host-based IDS (HIDS) Network-based IDS (NIDS) ◆ IDSes cooperate with each other or with an analyzer IDS analyzer distributed system server

3. Threats against the DIDS  Active attacks ◆ Directly take actions against IDSes by Sending malicious packets to network ports used by IDSes modifying IDS policy files or terminating IDS processes  Passive attacks ◆ Wait until IDSes read data including malicious code by Sending malicious packets to monitored servers Changing attributes of monitored files server IDS active attack monitor passive attack

4. Traditional approach: Isolated monitoring  Isolates NIDSes from servers physically ◆ Using NIDS hosts and a back-end switch  NIDS hosts monitor packets by port mirroring ◆ NIDS hosts are connected to mirroring ports in a front-end switch ◆ The front-end switch duplicates and forwards packets Internet back-end switch front-end switch server host NIDS host DIDS mirroring port

5. Internet back-end switch front-end switch server host NIDS host DIDS Security of isolated monitoring  Prevents active attacks ◆ The attacker cannot attack NIDS hosts using mirroring ports Mirroring ports are only for monitoring  Confines the impact of passive attacks to within the DIDS ◆ The attacker cannot access the outside of the DIDS ◆ Important because preventing passive attacks is difficult mirroring port

6. Problems in isolated monitoring  Need additional hardware ◆ Lots of machines for NIDSes ◆ A back-end switch ◆ A front-end switch with port mirroring  Support only NIDSes ◆ Legacy HIDSes do not support monitoring of remote server hosts ◆ Achieving secure monitoring of remote server hosts from HIDS hosts is difficult

7. Our approach: HyperSpector  Virtual distributed monitoring environment ◆ IDS VM and server VM Isolate each other without additional hardware The IDS VM can monitor the server VM ◆ A virtual network Connects the IDS VMs Isolated from a network used by servers server VM server VM server VM server VM virtual network IDS VM IDS VM IDS VM IDS VM DIDS

8. Inter-VM monitoring mechanisms  Requirements ◆ Interfaces to legacy IDSes ◆ Secure monitoring between VMs  HyperSpector provides three mechanisms ◆ Software port mirroring (for packet capturing) ◆ Inter-VM disk mounting (for file system checking) ◆ Inter-VM process mapping (for process checking)

9. Software port mirroring  Virtual switch ◆ Achieves port mirroring by software ◆ Connects its mirroring port to the IDS VM Using a virtual network interface (VNI) ◆ Duplicates and forwards packets to the IDS VM virtual switch virtual switch NIDS BPF device VNI server VMIDS VM VMM mirroring port outside

10. Inter-VM disk mounting  Inter-VM disk mounter ◆ Mounts the file system of the server VM on the IDS VM As a shadow file system ◆ Forwards requests to a shadow file system to the server VM Using VMM interfaces inter-VM disk mounter HIDS server VMIDS VM VMM interface file system shadow file system read

11. Inter-VM process mapping  Inter-VM process mapper ◆ Maps the processes in the server VM to the IDS VM As shadow processes ◆ Forwards Requests to shadow processes to the server VM Notifications from the server VM to HIDSes –Using VMM interfaces inter-VM process mapper HIDS server VMIDS VM VMM interface server process ptrace shadow process wakeup

12. server VM server VM server VM server VM virtual network IDS VM IDS VM IDS VM IDS VM Security of HyperSpector  Prevents active attacks ◆ From the server VMs ◆ From hosts outside the DIDS  Confines the impact of passive attacks ◆ The IDS VM cannot attack the server VM ◆ The IDS VM cannot attack hosts outside the DIDS DIDS server VM server VM server VM server VM IDS VM IDS VM IDS VM IDS VM

13. Security of the inter-VM monitoring mechanisms  Secure, because ◆ The server VM cannot use inter-VM monitoring mechanisms ◆ The IDS VM cannot interfere with the server VM Inter-VM monitoring mechanisms are only for monitoring ◆ The IDS VM cannot send monitored information outside the DIDS Although it can view secret information of servers... server VM IDS VM IDS VM VMM modify outside hosts monitor request

14. Implementation  We have implemented HyperSpector in the FreeBSD kernel  IDS VM and server VM ◆ Based on our portspace The portspace virtualizes only a network system, file system, and processes ◆ Secure enough We assume the kernel and the base system are not exploitable kernel base system IDS VMserver VM VMM net fs

15. Implementation of the VMM  Implemented efficiently in the kernel ◆ Virtual switch Maps a network interface of the server VM to the IDS VM in a read-only manner ◆ Inter-VM disk mounter Mounts the file system of the server VM on the IDS VM read-only, using the modified union file system ◆ Inter-VM process mapper Makes the IDS VM share the processes of the server VM in a read-only manner

16. Experiments  We measured overhead of HyperSpector ◆ Experimental setup Snort, Tripwire, or truss in the IDS VM thttpd in the server VM ApacheBench in the client host ◆ Hardware 2 PCs (3.0 GHz Pentium 4, 1 GB of memory, Intel Pro/100+) 100Base-T network switch IDS VM IDS VM server VM server VM client host server host

17. Snort  Monitors packets from ApacheBench to thttpd ◆ We measured the throughput of thttpd ◆ For comparison The base system Isolated monitoring  Maximum overhead ◆ 7.5% slower than the base system ◆ 7% slower than isolated monitoring (over 2 KB file size) 30% in 0 KB file size

18. Tripwire  Checks the integrity of the whole file system ◆ 54,885 objects ◆ We measured the time of the integrity check altering the file change rate ◆ For comparison The base system  Overhead ◆ 17 to 26% slower than the base system

19. Truss  Traces system calls issued by thttpd ◆ We measured the throughput of thttpd Using ApacheBench ◆ For comparison The base system  Overhead ◆ 0.8 to 7.3% slower than the base system

20. Related work  ReVirt [Dunlap’02], Livewire [Garfinkel’03] ◆ Enable IDSes to monitor servers running in a VM The VM protects IDSes from active attacks via servers ◆ Do not consider other attacks against IDSes  Backdoors [Bohra’04] ◆ Enables isolated monitoring for HIDSes Using programmable NICs to monitor server state ◆ Needs much hardware ◆ Insecure because HIDS hosts are network-reachable These need to develop specialized IDSes

21. Conclusion  We proposed HyperSpector, which ◆ Isolates IDSes from servers without additional hardware Using IDS VMs, server VMs, and a virtual network ◆ Provides secure Inter-VM monitoring mechanisms: Software port mirroring, inter-VM disk mounting, and inter-VM process mapping ◆ Prevents active attacks and confines the impact of passive attacks to within the DIDS

22. Future work  Support for active monitoring ◆ Needs a mechanism to securely send probe messages to servers  Support for DoS attacks ◆ Needs to allocate sufficient resources to the IDS VM even under overload  Automatic detection of compromised HyperSpector ◆ Monitoring resource usage may help