1. 1 Introduction to Honeypot, Denial-of- Service, and Rootkit Cliff C. Zou CAP6135 Spring, 2011

2. Acknowledgement Some contents on honeypot are from  http://staff.washington.edu/dittrich/talks/aro- honeynets.ppt http://staff.washington.edu/dittrich/talks/aro- honeynets.ppt Some figures on DDoS are from  http://www.cisco.com/web/IT/events/pdf/iin2005/dist ributed_denial.pdf http://www.cisco.com/web/IT/events/pdf/iin2005/dist ributed_denial.pdf 2

3. 3 What Is a Honeypot? Abstract definition: “A honeypot is an information system resource whose value lies in unauthorized or illicit use of that resource.” (Lance Spitzner) Concrete definition: “A honeypot is a faked vulnerable system used for the purpose of being attacked, probed, exploited and compromised.”

4. 4 Example of a Simple Honeypot Install vulnerable OS and software on a machine Install monitor or IDS software Connect to the Internet (with global IP) Wait & monitor being scanned, attacked, compromised Finish analysis, clean the machine

5. 5 Benefit of Deploying Honeypots Risk mitigation:  Lure an attacker away from the real production systems (“easy target“). IDS-like functionality:  Since no legitimate traffic should take place to or from the honeypot, any traffic appearing is evil and can initiate further actions.

6. 6 Benefit of Deploying Honeypots Attack analysis:  Find out reasons, and strategies why and how you are attacked.  Binary and behavior analysis of capture malicious code Evidence:  Once the attacker is identified, all data captured may be used in a legal procedure. Increased knowledge

7. 7 Honeypot Classification High-interaction honeypots  A full and working OS is provided for being attacked  VMware virtual environment  Several VMware virtual hosts in one physical machine Low-interaction honeypots  Only emulate specific network services  No real interaction or OS  Honeyd Honeynet/honeyfarm  A network of honeypots

8. 8 Low-Interaction Honeypots Pros:  Easy to install (simple program)  No risk (no vulnerable software to be attacked)  One machine supports hundreds of honeypots, covers hundreds of IP addresses  Can distinguish most attacks on the same port Cons:  No real interaction to be captured  Limited logging/monitor function  Hard to detect unknown attacks; hard to generate filters  Easily detectable by attackers

9. Emulation of Services 9 QUIT* ) echo -e "221 Goodbye.\r" exit 0;; SYST* ) echo -e "215 UNIX Type: L8\r" ;; HELP* ) echo -e "214-The following commands are recognized (* =>'s unimplemented).\r" echo -e " USER PORT STOR MSAM* RNTO NLST MKD CDUP\r" echo -e " PASS PASV APPE MRSQ* ABOR SITE XMKD XCUP\r" echo -e " ACCT* TYPE MLFL* MRCP* DELE SYST RMD STOU\r" echo -e " SMNT* STRU MAIL* ALLO CWD STAT XRMD SIZE\r" echo -e " REIN* MODE MSND* REST XCWD HELP PWD MDTM\r" echo -e " QUIT RETR MSOM* RNFR LIST NOOP XPWD\r" echo -e "214 Direct comments to ftp@$domain.\r" ;; USER* )

10. 10 High-Interaction Honeypots Pros:  Real OS, capture all attack traffic/actions  Can discover unknown attacks/vulnerabilites  Can capture and anlayze code behavior Cons:  Time-consuming to build/maintain  Time-consuming to analysis attack  Risk of being used as stepping stone  High computer resource requirement

11. 11 Honeynet A network of honeypots High-interaction honeynet  A distributed network composing many honeypots Low-interaction honeynet  Emulate a virtual network in one physical machine  Example: honeyd

12. Gen II Honeynet 12

13. Data Control 13 Prevent a honeypot being used by attackers to attack others (legal/ethnical issues)

14. 14 Honeypot-Aware Botnet [Zou’07] Honeypot is widely used by defenders  Ability to detect unknown attacks  Ability to monitor attacker actions (e.g., botnet C&C) Botnet attackers will adapt to honeypot defense  When they feel the real threat from honeypot  We need to think one step ahead

15. 15 Honeypot Detection Principles Hardware/software specific honeypot detection  Detect virtual environment via specific code  E.g., time response, memory address  Detect faculty honeypot program  Case by case detection Detection based on fundamental difference  Honeypot defenders are liable for attacks sending out  Liability law will become mature  It’s a moral issue as well  Real attackers bear no liability  Check whether a bot can send out malicious traffic or not

16. 16 Detection of Honeypot Bot Infection traffic  Real liability to defenders  No exposure issue: a bot needs to do this regardless Other honeypot detection traffic  Port scanning, email spam, web request (DoS?) C&C bot Sensor (secret) 1 malicious traffic 2 Inform bot’s IP 3 Authorize

17. 17 Two-stage Reconnaissance to Detect Honeypot in Constructing P2P Botnets Fully distributed  No central sensor is used  Could be fooled by double-honeypot  Counterattack is presented in our paper Lightweighted spearhead code  Infect + honeypot detection  Speedup UDP-based infection Host A spearhead Host B request main-force spearhead Host C 1 3 2

18. 18 Defense against Honeypot-Aware Attacks Permit dedicated honeypot detection systems to send out malicious traffic  Need law and strict policy Redirect outgoing traffic to a second honeypot  Not effective for sensor-based honeypot detection Figure out what outgoing traffic is for honeypot detection, and then allow it  It could be very hard Neverthless, honeypot is still a valuable monitoring and detection/defense tool

19. 19 Distributed Denial of Service (DDoS) Attack Send large amount of traffic to a server so that the server has no resource to serve normal users Attacking format:  Consume target memory/CPU resource  SYN flood (backscatter paper presented before)  Database query…  Congest target Internet connection  Many sources attack traffic overwhelm target link  Very hard to defend

20. 20 Why hard to defined DDoS attack? Internet IP protocol has no built-in security  No authentication of source IP  SYN flood with faked source IP  However, IP is true after connection is setup Servers are supposed to accept unsolicited service requests Lack of collaboration ways among Internet community  How can you ask an ISP in another country to block certain traffic for you?

21. 21 DDoS Defenses Increase servers capacity  Cluster of machine, Multi-CPUs, larger Internet access Use Internet web caching service  E.g., Akamai Defense Methods (many in research stage)  SYN cookies (http://en.wikipedia.org/wiki/SYN_cookies)http://en.wikipedia.org/wiki/SYN_cookies  SOS  IP traceback

22. 22

23. 23

24. 24 SYN Cookies SYN flood attack  Fill up server’s SYN queue  Property: attacker does not respond to SYN/ACK from victim. Defense  Fact: normal client responds to SYN/ACK  Remove initial SYN queue  Server encode info in TCP seq. number  Use it to reconstruct the initial SYN

25. 25 DoS spoofed attack defense: IP traceback Suppose a victim can call ISPs upstream to block certain traffic SYN flood: which traffic to block? IP traceback:  Find out the real attacking host for SYN flood  Based on large amount of attacking packets  Need a little help from routers (packet marking)

26. 26 SOS: Secure Overlay Service Central Idea:  Use many TCP connection respondent machines  Only setup connections relay to server  Identity of server is secrete

27. The Evolution of Malware Malware, including spyware, adware and viruses want to be hard to detect and/or hard to remove Rootkits are a fast evolving technology to achieve these goals  Cloaking technology applied to malware  Not malware by itself  Example rootkit-based viruses: W32.Maslan.A@mm, W32.Opasa@mm Rootkit history  Appeared as stealth viruses  One of the first known PC viruses, Brain, was stealth  First “rootkit” appeared on SunOS in 1994  Replacement of core system utilities (ls, ps, etc.) to hide malware processes

28. Cloaking Modern rootkits can cloak:  Processes  Services  TCP/IP ports  Files  Registry keys  User accounts Several major rootkit technologies  User-mode API filtering  Kernel-mode API filtering  Kernel-mode data structure manipulation  Process hijacking Visit www.rootkit.com for tools and informationwww.rootkit.com

29. Attack user-mode system query APIs Con: can be bypassed by going directly to kernel- mode APIs Pro: can infect unprivileged user accounts Examples: HackerDefender, Afx Taskmgr.exe Ntdll.dll Explorer.exe, Malware.exe, Winlogon.exe Rootkit Explorer.exe, Winlogon.exe user mode kernel mode User-Mode API Filtering

30. Attack kernel-mode system query APIs Cons:  Requires admin privilege to install  Difficult to write Pro: very thorough cloak Example: NT Rootkit Taskmgr.exe Ntdll.dll user mode user mode kernel mode kernel mode Rootkit Explorer.exe, Winlogon.exe Explorer.exe, Malware.exe, Winlogon.exe Kernel-Mode API Filtering

31. Also called Direct Kernel Object Manipulation (DKOM) Attacks active process data structure  Query API doesn’t see the process  Kernel still schedules process’ threads Cons:  Requires admin privilege to install  Can cause crashes  Detection already developed Pro: more advanced variations possible Example: FU Explorer.exeMalware.exeWinlogon.exe Active Processes Kernel-Mode Data Structure Manipulation

32. Hide inside a legitimate process Con: doesn’t survive reboot Pro: extremely hard to detect Example: Code Red Explorer.exe Malware Process Hijacking

33. Detecting Rootkits All cloaks have holes  Leave some APIs unfiltered  Have detectable side effects  Can’t cloak when OS is offline Rootkit detection attacks holes  Cat-and-mouse game  Several examples  Microsoft Research Strider/Ghostbuster  RKDetect  Sysinternals RootkitRevealer  F-Secure BlackLight

34. Perform a directory listing online and compare with secure alternate OS boot (see http://research.microsoft.com/rootkit/ )http://research.microsoft.com/rootkit/  Offline OS is Windows PE, ERD Commander, BartPE dir /s /ah * > dirscan.txt windiff dirscanon.txt dirscanoff.txt This won’t detect non-persistent rootkits that save to disk during shutdown Simple Rootkit Detection

35. RootkitRevealer Rootkit Windows API Raw file system, Raw Registry hive Filtered Windows API omits malware files and keys Malware files and keys are visible in raw scan RootkitRevealer RootkitRevealer (RKR) runs online RKR tries to bypass rootkit to uncover cloaked objects  All detectors listed do the same  RKR scans HKLM\Software, HKLM\System and the file system  Performs Windows API scan and compares with raw data structure scan

36. Demo HackerDefender  HackerDefender before and after view of file system  Detecting HackerDefender with RootkitRevealer

37. Unless you have specific uninstall instructions from an authoritative source: Don’t rely on “rename” functionality offered by some rootkit detectors  It might not have detected all a rootkit’s components  The rename might not be effective Reformat the system and reinstall Windows! Dealing with Rootkits