1. 11 Active Botnet Probing to Identify Obscure Command and Control Channels G Gu, V Yegneswaran, P Porras, J Stoll, and W Lee - on Annual Computer Security Applications Conference 2009 (ACSAC 2009 ) Reporter: 高嘉男 Advisor: Chin-Laung Lei 2010/3/15

2. 2 Outline Introduction Problem statement & assumptions Active botnet probing: architecture & algorithms Experiments with BotProbe Conclusion

3. 3 Introduction Botnet C&C channel: existing protocols ◦ IRC, HTTP & P2P Botnet detection: passive ◦ Signature-based detection ◦ Honeypot-based detection ◦ Behavior-based botnet detection Contemporary IRC botnet ◦ Obfuscated IRC messages ◦ Small sizes ◦ Infrequent C&C interactions

4. 4 Active Method Collect evidence actively Assume there is only one round of (obscure) chat-like botnet C&C interaction from one bot, can we still detect the bot with a high probability?

5. 5 Key Observations Botnet C&C interaction has a clear command-response pattern ◦ A bot will behave deterministically to replayed commands Bots are preprogrammed to respond to the set of commands they receive ◦ Bots have limited tolerance for typographical errors in conversations

6. 6 Adversary Assumption A bot should respond when it receives a predefined command in a reasonable time Message response ◦ IRC PRIVMSG message Activity response ◦ Scan response ◦ Third-party response ◦ Spam response

7. 7 Architecture Design

8. 8 Active Probing Techniques

9. 9 Active Probing Techniques (Cont’d) P0 (Explicit-Challenge-Response) ◦ Reverse Turing test ◦ Request the user to visit a website to read and translate a CAPTCHA P1 (Session-Replay-Probing) ◦ Replay the same application command to the client several times

10. 10 Active Probing Techniques (Cont’d) P2 (Session-Byte-Probing) ◦ The BotProbe monitor randomly permutes certain bytes of the application command P3 (Client-Replay-Probing) ◦ Register a new user into the channel ◦ Send the observed command(s) to the selected client P4 (Man-In-The-Middle-Probing) ◦ Intercept the new command and launch a man-in-the-middle-like chat message injection

11. 11 Turing-Test-Hypothesis Algorithm Perform one or more rounds of P0 probing H 1 : the hypothesis “botnet C&C” H 0 : the hypothesis “normal chat” Binary random variable D: whether or not we observe a wrong reply for a challenge from the client (D = 1: an incorrect reply) θ 1 = Pr( D=1 | H 1 ), θ 0 = Pr( D=1 | H 0 ) θ 1 ≒ 1, θ 0 ≒ 0 α : false positive rate, β : false negative rate n : rounds of probing Define

12. 12 Turing-Test-Hypothesis Algorithm (cont’d) Threshold random walk (TRW) ◦ Walk starts from origin(0) ◦ Walk goes up with length ln( θ 1 / θ 0 ) if D i = 1 ◦ Walk goes down with length ln(1- θ 1 /1- θ 0 ) if D i = 0 After n rounds ◦ If Λ n > ln(1- β / α ): H 1 is true, it is a botnet C&C ◦ If Λ n < ln( β /1- α ): H 0 is true, it is a normal IRC dialog ◦ If else: additional rounds of testing

13. 13 Single-Binary-Response-Hypothesis Algorithm Perform one or more rounds of P1 probing D: whether or not a response from the client is observed Iterate the TRW process at different scales depending on the responses Multiple different types of responses corresponding to the same command ◦ Choose the one that provides highest confidence (walks a largest step)

14. 14 Interleaved-Binary-Response- Hypothesis Algorithm Perform one or more rounds of interleaved P1 and P2 probing D = 1: the observation of a response from the replayed packets and no response from modified packets Bots ◦ Respond to replayed packets reliably ◦ Do not recognize the modified command Human ◦ Respond to a message with typographical error ◦ How normal users may respond to two replayed IRC messages?

15. 15 Evaluating User Disturbance The degree of disturbance ◦ The number of rounds (packets modified/replayed) To produce a botnet C&C declaration To produce a human user IRC channel declaration

16. 16 Evaluating User Disturbance (cont’d)

17. 17 BotProbe: an Active Botnet Probing System

18. 18 Test the False Negative Rate How many bot C&Cs are missed by BotProbe? Execute the bot in Windows XP (VMware) Monitor with BotProbe on Linux Three classes of real-world IRC bots ◦ Open-source bots with obfuscated communication  Spybot ◦ Bot binaries with cleartext communication  Phatbot, Rbot, Rxbot, Sdbot ◦ Bot binaries with obfuscated communication  W32.Wargbot, Trojan.Dropper.Sramler.C

19. 19 Test the False Negative Rate (cont’d) Parameters of testing algorithm ◦ θ 1 =0.99, θ 0 =0.15, α (FP)=0.001, β (FN)=0.01 ◦ θ 0 scan =0.01, θ 0 3rd-party-access =0.02

20. 20 Test the False Negative Rate (cont’d) W32.Wargbot ◦ Put an encrypted command in the IRC TOPIC message for bots to execute Trojan.Dropper.Sramler.C

21. 21 Test the False Positive Rate How frequently could normal chatting sessions be mislabeled as botnet C&C Study design ◦ Human users periodically sent messages that simulate the effect of botnet probing to real users at diverse channels Test on two different platforms ◦ IRC & mebbo.com

22. 22 Test the False Positive Rate (cont’d) Study design ◦ Design six different questions to test 123 different users ◦ Questions  “what’s up” “nice weather” “you like red?” “how may I help you?” “English only! I play nice fun” ◦ Modified questions  “waat’s up” “noce weather” “aou like red?” “Bow may I help you?” “Eaglish only! I play nice fun” ◦ Turing test messages  “what’s 3+6=?”

23. 23 Test the False Positive Rate (cont’d)

24. 24 Conclusion The first feasibility study of the use of active techniques in botnet detection ◦ Collect evidence actively ◦ Shorten the detection time A hypothesis testing framework & a prototype system implementation ◦ Separates deterministic botnet communication from human conversations effectively

25. 25 Reference G Gu, V Yegneswaran, P Porras, J Stoll, and W Lee, “Active Botnet Probing to Identify Obscure Command and Control Channels.” in Annual Computer Security Applications Conference, 2009.