Log Analysis and Intrusion Detection By Srikrishna Gudavalli Venkata Naga Vamsi Krishna Ravi Kiran Yellepeddy

Log Analysis (Windows And linux) What is log analysis? Describes an event (or) process activity in detail on the system. Examples : user authentication event log ftp authentication.

Setup for LogAnalysis Application Log Specific to particular application. eg:MS word,Windows Media Player Security Log Specifically logs all the security features. System Log Logs all the system related activities.

Log Files Lokasi bergantung, umumnya pada /var/log Contoh /var/log/mail.log /var/log/messages /var/log/daemon.log /var/log/apache/access-log /var/log/apache/error-log /var/adm/utmp /var/adm/wtmp

5 Log Analysis and Correlation Syslogs, messages logs, other Unix host logs Security/Auth Log Mar 9 13:07:49 nile in.telnetd[1315]: connect from Mar 9 13:09:24 nile in.rlogind[1321]: connect from Mar 9 13:09:27 nile in.ftpd[1326]: connect from Mar 9 13:09:28 nile in.rshd[1329]: connect from Mar 9 13:09:28 nile in.telnetd[1333]: connect from Mar 9 13:09:31 nile in.fingerd[1334]: connect from Mar 9 13:12:13 nile in.fingerd[1352]: connect from Mar 9 13:12:13 nile in.rlogind[1357]: connect from Mar 9 13:12:14 nile in.rshd[1360]: connect from Mar 9 13:12:16 nile in.telnetd[1365]: connect from Mar 9 13:12:18 nile in.ftpd[1368]: connect from Mar 9 13:15:23 nile in.ftpd[1382]: connect from Mar 9 13:15:24 nile in.telnetd[1384]: connect from Mar 9 13:15:27 nile in.rshd[1396]: connect from Mar 9 13:15:28 nile in.rlogind[1398]: connect from Mar 9 13:15:29 nile in.fingerd[1400]: connect from Mar 9 13:26:43 nile login: ROOT LOGIN ON tty1 Mar 9 13:37:15 nile in.ftpd[1447]: connect from Mar 9 13:37:44 nile in.fingerd[1448]: connect from Mar 9 17:17:19 nile in.telnetd[1521]: connect from Mar 9 17:17:26 nile login: LOGIN ON 0 BY pstephen FROM 43.detroit-16-17rs.mi.dial-access.att.net Mar 9 17:50:13 nile in.ftpd[1556]: connect from Mar 10 11:12:02 nile in.ftpd[8929]: connect from Mar 10 11:13:07 nile in.ftpd[8965]: connect from

6 Log Analysis and Correlation TCPDump logs 11:30: eth0 nile.ftp:. 1:1(0) ack 1 win 4288 (DF) 11:30: eth0 > arp who-has ubr01-a-rtr.aubrnh01.mi.comcast.net tell nile (0:0:86:54:50:5b) 11:30: eth0 nile.1025: /2/2 PTR pcp pcs.aubrnh01.mi.comcast.net. (174) (DF) 11:30: eth0 nile.1025: /2/2 A pcp pcs.aubrnh01.mi.comcast.net (151) (DF) 11:30: eth0 nile.1025: /2/2 PTR pcp pcs.aubrnh01.mi.comcast.net. (174) (DF) 11:30: eth0 > nile.ftp > pcp pcs.aubrnh01.mi.comcast.net.17697: P 1:80(79) ack 1 win (DF) [tos 0x10] 11:30: eth0 nile.ftp: P 1:16(15) ack 80 win 4209 (DF) 11:30: eth0 > nile.ftp > pcp pcs.aubrnh01.mi.comcast.net.17697:. 80:80(0) ack 16 win (DF) [tos 0x10]

7 Log Analysis and Correlation Intrusion Detection Log (RealSecure)

8 Log Analysis and Correlation Intrusion Detection Log (SNORT Summary) Apr 16 02:45:37 lisa snort[7483]: IDS13/portmap-request-mountd: :1372 -> :111 Apr 16 07:17:06 lisa snort[7483]: IDS128/web-cgi-phf: : > :80 Apr 16 14:54:20 lisa snort[7483]: IDS171/Ping zeros: > Apr 16 14:54:20 lisa snort[7483]: IDS171/Ping zeros: > Apr 16 14:54:20 lisa snort[7483]: IDS171/Ping zeros: > Apr 17 06:02:32 lisa snort[8255]: IDS198/SYN FIN Scan: :0 -> :111 Apr 17 06:02:32 lisa snort[8255]: IDS198/SYN FIN Scan: :0 -> :111 Apr 17 09:45:28 lisa snort[8255]: IDS198/SYN FIN Scan: :0 -> :111 Apr 19 08:00:19 lisa snort[3515]: IDS/DNS-version-query: :1723 -> :53 Apr 20 01:26:00 lisa snort[3515]: IDS212/dns-zone-transfer: :4075 -> :53 Apr 20 03:49:38 lisa snort[3515]: IDS/DNS-version-query: :4349 -> :53 Apr 20 03:49:39 lisa snort[3515]: IDS/DNS-version-query: :4350 -> :53 Apr 20 21:48:55 lisa snort[12353]: IDS246/large-icmp: > Apr 20 22:48:13 lisa snort[12632]: IDS159/Ping Microsoft Windows: > Apr 20 23:00:33 lisa snort[12657]: IDS171/Ping zeros: > Apr 21 11:01:27 lisa snort[12777]: IDS/DNS-version-query: :4039 -> :53 Apr 21 11:01:28 lisa snort[12777]: IDS/DNS-version-query: :4044 -> :53 Apr 22 08:36:29 lisa snort[743]: IDS/DNS-version-query: :1368 -> :53 Apr 22 08:36:29 lisa snort[743]: IDS/DNS-version-query: :1328 -> :53 Courtesy of The Honeynet Project

9 Log Analysis and Correlation Intrusion Detection Log (SNORT Raw Log) [**] WEB-MISC 403 Forbidden [**] 07/29-23:59: :0:C5:75:67:2C -> 0:AA:0:B7:E9:56 type:0x800 len:0x :80 -> :1550 TCP TTL:43 TOS:0x0 ID:22555 IpLen:20 DgmLen:568 DF ***AP*** Seq: 0x85B19798 Ack: 0x4E439F5C Win: 0x7D78 TcpLen: F 31 2E F 72 HTTP/ For E 0D 0A A 20 4D 6F bidden..Date: Mo 6E 2C A 75 6C n, 30 Jul A A D 54 0D 0A :58:58 GMT..Ser A F 31 2E 33 2E ver: Apache/ E D 6F 64 5F (Unix) mod_ss 6C 2F 32 2E 38 2E F E C 2F l/2.8.3 OpenSSL/ 30 2E 39 2E D 6F 64 5F C 2F 0.9.6a mod_perl/ 31 2E D 6F 64 5F 67 7A F 31 2E 1.25 mod_gzip/ E E F 34 2E 30 2E a PHP/ D 0A 43 6F 6E 6E F 6E 3A Connection: c 6C 6F D 0A 43 6F 6E E 74 2D lose..Content-Ty A F D 6C 3B pe: text/html; c D F 2D D harset=iso D 0A 0D 0A 3C F <!DOCTYPE H 54 4D 4C C D 2F 2F 49 TML PUBLIC "-//I F 2F D 4C E ETF//DTD HTML F 2F 45 4E 22 3E 0A 3C D 4C 3E 3C 48 0//EN">. <H E 0A 3C C 45 3E EAD> F E 3C 2F C 45 Forbidden</TITLE 3E 0A 3C 2F E 3C 42 4F E 0A >.. 3C E 46 6F E 3C 2F 48 Forbidden</H 31 3E 0A 59 6F F 6E >.You don't hav D F 6E F 20 e permission to F D E 2F access /cgi-bin/ C 65 2F C 65 2E adcycle/adcycle A 6F 6E cgi.on this serv E 3C 50 3E 0A 3C E 0A 3C er...<ADD E F 31 2E 33 2E RESS>Apache/ Server at the F 72 6E E 63 6F 6D babycorner.com P 6F C 2F E ort 80 0A 3C 2F 42 4F E 3C 2F D 4C 3E 0A..

10 Log Analysis and Correlation Correlating data from multiple sources –Normalizing Same events may have different names depending upon the source –Translating IDS codes »Cisco NetRanger: 4052 »ISS RealSecure: Chargen_Denial_of_Service Use to build a chain of evidence

11 Log Analysis and Correlation Correlating data from multiple sources –Deconfliction Same event shows up multiple times with same names –Certain types of denial of service attacks –Some penetration attacks »Use care not to remove individual steps in an attack scenario Same event repeated so rapidly that the logging device reports a large number of the same event in a very short (sometimes sub-second) period of time Multiple rapid events that make an attack scenario such as a port scan Deconflicted events are used with normalized data to create an event timeline

12 Log Analysis and Correlation Correlating data from multiple sources –Creating chain of evidence and event timelines Using deconflicted and normalized events on multiple data sources, chart the chain of events into an event timeline –Carefully note the timebase of various data sources and correct to a common timebase –Note events and attack scenarios – correlate connected events into scenarios Document every assumption with evidence and, if possible, corroboration using both forensic and traditional investigation

13 Log Analysis and Correlation Forensic handling of deleted or modified logs –Useful only in certain types of systems Recovering deleted logs –System must support recovery of ambient data Recovering altered logs –Logging source must delete old log and create a new one when the log is altered –System must support recovery of ambient data

Web Server Log Analysis

SD'98 (c) David Strom, Inc.15 Different types of log files Access Error Referral Other

SD'98 (c) David Strom, Inc.16 Access logs Domain name Date, time Server command processed and result URL of visitor Bytes transmitted

SD'98 (c) David Strom, Inc.17 Sample access log data rm258.fav.usu.edu [31/May/1995:09:03: ] "GET /NEI.html HTTP/1.0" rm258.fav.usu.edu [31/May/1995:09:03: ] "GET /xculture/nei/nei.html HTTP/1.0" rm258.fav.usu.edu [31/May/1995:09:03: ] "GET /gifs/sedlbutton.gif HTTP/1.0" [31/May/1995:09:20: ] "GET /RELs.html HTTP/1.0" Leslie-Francis.tenet.edu [31/May/1995:09:36: ] "GET / HTTP/1.0" ls973.ulib.albany.edu [31/May/1995:09:40: ] "GET /viii1.html HTTP/1.0"

SD'98 (c) David Strom, Inc.18 Errors reported in your logs Clients that time out (or leave in frustration!) Scripts that don’t produce any output Server bugs User authentication or configuration problems

SD'98 (c) David Strom, Inc.19 Sample error log data [Thu May 30 07:25: ] send timed out for bamberg.sedl.org [Thu May 30 07:57: ] send timed out for kenya.sedl.org [Thu May 30 08:23: ] send timed out for ppp092.kyoto- inet.or.jp [Thu May 30 09:15: ] access to /usr/local/www/htdocs/scimath/compass/vol03 failed for , reason: File does not exist [Thu May 30 09:57: ] send timed out for dd compuserve.com [Thu May 30 10:47: ] read timed out for ncia110b.ncia.net

SD'98 (c) David Strom, Inc.20 Referral logs Who links to your site? Who downloads your pages?

SD'98 (c) David Strom, Inc.21 Sample referral log data ->/change/welcome.html ->/welcome.html ->/policy/networks/toc.html ->/policy/networks/toc.html - >/resources/SCIMAST/announcement.html ->/policy/networks/toc.html - >/welcome.html

SD'98 (c) David Strom, Inc.22 Common log format Output by most standard servers Needed by most third-party log analyzers hoohoo.ncsa.uiuc.edu/docs/setup/httpd/Overview.html

SD'98 (c) David Strom, Inc.23 Extended/custom log formats Log whatever you wish in whatever order you wish Useful if you will read them regularly! But can’t work with the analyzers Now in IIS v4, NSCP v3, others.

SD'98 (c) David Strom, Inc.24 What you can learn from your log files Hits per day Domain origins The path people take in and around your web Problem areas

SD'98 (c) David Strom, Inc.25 HITS (How Idiots Track Success) Nobody uses this word anymore Doesn’t really measure individual users, just access Catching servers and proxies mess up these statistics

SD'98 (c) David Strom, Inc.26 Domain origins Where users are coming from -- sometimes Just because they are from ibm.net doesn’t mean they work at IBM! Forgotten accounts, friends and family using the account Hacked user names Proxies don’t help here either

SD'98 (c) David Strom, Inc.27 The path people take in and around your web Search engines help sometimes Which search site was the most popular front door Who links to you and why Is there a pattern or a random walk?

SD'98 (c) David Strom, Inc.28 Problem areas to deal with Broken links (locally) Broken outbound links Time outs (sunspots?)

SD'98 (c) David Strom, Inc.29 What you can’t learn from your logs Who are these people, anyway? –No specific user names –Is it a bot or a real human? How long did they view a page? –Most people don’t spend much time on your web –Where did they go visit next?

SD'98 (c) David Strom, Inc.30 What technologies are available? Built-in analyzer tools Sites that capture user info Secure sites with registration Build your own from perl Third-party tools

SD'98 (c) David Strom, Inc.31 Built-in tools WebSite, website.ora.com IIS with Site Server, Netscape servers, Easy to use but limited

SD'98 (c) David Strom, Inc.32 WebSite Professional v2 Win NT, 95 Best web server for learning about logs, best docs QuickStats module for instant analysis: –single report but nice set of information –shows today, last two days requests and unique hosts –IP addresses of visitors, average requests/hour

SD'98 (c) David Strom, Inc.33 IIS Site Server NT Server v4 w/SP3 only Lots of preconfigured reports Two versions, Express and Full (customized reports) backoffice.microsoft.com/products/siteserv er/express/

SD'98 (c) David Strom, Inc.34 Netscape v3 web servers Various NT, Unix versions Reports for a few variables but nothing too extensive Best to use a third-party tool here

SD'98 (c) David Strom, Inc.35 Sites that capture user info WebCounter, -- third-party hit counter Someone else does the programming and debugging But beyond your control

SD'98 (c) David Strom, Inc.36 Secure sites with registration You know your users But many won’t register, or forget their passwords Requires scripting, database integration, more maintenance

SD'98 (c) David Strom, Inc.37 Build your own from perl Needs some in-house support Works best with Unix-based webs Examples: –refstats, members.aol.com/htmlguru/refstats.html –surfreport, bienlogic.com/SurfReport/

SD'98 (c) David Strom, Inc.38 Third-party tools WebTracker, WebTrends, net.Genesis, MarketWave, IIS Assistant,

SD'98 (c) David Strom, Inc.39 Third-party tools (con’t) Can make very pretty reports Customizable Make sure they support your particular log format Not that expensive, mostly run on Windows