1. LMA: Log Mail Analyzer Maurizio Aiello maurizio.aiello@ieiit.cnr.it National Research Council Institute of Electronics and Telecommunications and Information Engineering ( IEIIT ) http://sourceforge.net/lma

2. Free software project LMA: Log Mail Analyzer What can be performed with Log File Analysys? –User’s request –Normal debugging operations –Help for worm detection Why do we need a tool for log mail analysis? Mainly, avoiding headache Speeding up operation

3. Postfix architecture Why are log files so complex? –Modularity –Log = Debug –…

4. Interesting fields What information do we need about an e-mail transaction? Using hash QID (queue identifier) we retrieve value for each field above TimestampIp clientMail FromRcpt toStatus

5. Postfix :remote client to local user

6. E-mail translation Retrieving info on a mail: Find its QID Search lines related to that QID Reconstruct transaction (Local-Local, L-Remote, R-L, R-R) LMA Module: Log-Translator Output: info file (plaintext)

7. Architectural issue Customization needs: –Network architecture –Antivirus server –…. File conf: –Whitelisting –Network selection –DB format, server type

8. Database generation To store e-mail transaction we support 2 options: Transactional db: MysqlBerkeley DB + query flexibility + engine power + LMA standalone program (no db engine required) - need to install engine- need to build engine - engine power and flexibility

9. Dbgenerator module With berkeleyDB we have to build db engine:

10. Database keys and values Database KeyValue Mail_db E-mail_number (progressive integer) Timestamp, ip, from, to, status Date_dbTimestamp IP_dbIp address Receiver_db“Rcpt to” recipient Sender_db“mail from” sender Sequence of e- mail_number

11. Database schema

12. Query engine and example To search through DB, LMA performs the following: Example: find all e-mails sent from aiello@ge.cnr.it: 1. search aiello@ge.cnr.it in Sender_db table aiello@ge.cnr.it 2. obtain a list of integer which are keys in mail table aiello@ge.cnr.it -> 27 | 45| 78| 3456| 8960 etc. 3. retrieve all the data about each e-mail 27 ->01-Jan- 2004|xxx.yyy.www.zzz|aiello@ge.cnr.it|jake@dot.com|250

13. Built-in query List all e-mail sent with the following characteristics: IP: from a particular IP FROM: with a given “mail from” field TO: to a particular recipient DATE: with ts_begin < timestamp < ts_final Sysman & Debugging OK.

14. Security? What about security? Worms use “direct” method to spread, scanning ports and exploiting vulnerabilities, or Use “indirect” way, for example using its own smtp engine or smtp server taken from User Agent settings.

15. Security aspects PC is infected by an indirect worm: we expect Lots of e-mail sent in a given time period; Different “mail from” field used by the same ip; Some abnormal mail repudiation by internet server. LMA birth: awk ' BEGIN { FS="[" } /client=/ { print $3 } ' < mail.log | sed s/]// | sort | uniq -c | sort -r

16. Another free project: Worm Poacher Project with aim to: study behaviour of e-mail client Detect anomalies Take the appropriate countermeasure

17. Statistical data mining Number of e-mails sent every 5m, 1h, 4h, 8h, 24h are calculated, plotted and analyzed

18. Baseline & statistichal Visual inspections and Baseline threshold analysis and alert raising: Baseline =   Calculated subtracting “inactivity period” Correlation between different time_slice (5m, 1h etc.) alerts to reduce false alarms.

19. Mail from Normally, client pc use few Mail from fields. Some worms change this field (stealthyness) Strange behaviour for a Pc? 80 different address in a day! As before baseline calculated statistically for each ip.

20. Reject analysis When a worm tries to spread fast, sometimes it chooses a random list of recipient (like jack@somedomain.com). jack@somedomain.com Probably a lot of these messages are rejected. Baseline calculation and threshold analysis.

21. Kind of analysys performed Global FlowSingle ip flow Number of e-mails sent XX Different mail from address XX Number of rejected mails XX

22. Single ip flow analysis Baseline calculated on each ip, instead of global traffic Single ip flow useful in big network (where signal/noise ratio is low). Performance problem and architectural issue (impossible to perform with dhcp, shared pc etc.)

23. Results

24. Worm decision

25. Future development Baseline dinamically updated Alarms generated by daemon SMTPsniffer. Reason: system independent from logfile format; can control any server. Extension to ports different from 25.