CIT 485: Advanced Cybersecurity Logs

Topics System logs Finding logs Syslog and syslog-ng Centralized logging Log monitoring SIEM

System Logs Logs record status and error conditions. Where do log messages come from? Kernel Accounting system System services Application and server software Logging methods: Software records own logs (apache, cron). UNIX software uses syslog service to manage logs. Windows software uses OS Event Log service. Network devices use syslog or SNMP traps.

Finding Logs On UNIX systems, most logs are stored under /var/log /var/adm Check syslog's configuration /etc/syslog.conf To find other logs, read startup scripts /etc/init.d/* and manuals for services started by scripts.

Security Logs Fewer than 1% of OS log entries relate to security. Similar percentages for most network devices, but 100% of firewall or IDS logs are relevant to security. Important security log files on Linux include auth.log, which records logins and su+sudo activity. wtmp, which records additional login data.

Finding Logs Log file Program Contents messages syslog Various program/kernel logs. auth.log su, ssh, login Authorization fail/success. lastlog login, xdm Logins, commands. wtmp login Login accounting data. acct/pacct kernel UNIX process accounting. Xorg.log X-Windows X-Windows failures/info.

Syslog Comprehensive logging system. Sorts messages by Frees programmers from managing log files. Gives sysadmins control over log management. Sorts messages by Sources Importance Routes messages to destinations Files Network Terminals

Syslog Components Syslog openlog, syslog, closelog logger Daemon that does actual logging. Additional daemon, klog, gets kernel messages. openlog, syslog, closelog C library routines to submit logs to syslog. logger User-level program to submit logs to syslog. Can use from shell scripts.

Example Syslog Messages Feb 11 10:17:01 localhost /USR/SBIN/CRON[1971]: (root) CMD ( run-parts --report /etc/cron.hourly) Feb 11 10:37:22 localhost -- MARK -- Feb 11 10:51:11 localhost dhclient: DHCPREQUEST on eth1 to 192.168.1.1 port 67 Feb 11 10:51:11 localhost dhclient: DHCPACK from 10.42.1.1 Feb 11 10:51:11 localhost dhclient: bound to 10.42.1.55 -- renewal in 35330 seconds. Feb 11 14:37:22 localhost -- MARK -- Feb 11 14:44:21 localhost mysqld[7340]: 060211 14:44:21 /usr/sbin/mysqld: Normal shutdown Feb 12 04:46:42 localhost sshd[29093]: Address 218.38.30.101 maps to ns.thundernet.co.kr, but this does not map back to the address - POSSIBLE BREAKIN ATTEMPT! Feb 12 04:46:44 localhost sshd[29097]: Invalid user matt from ::ffff:218.38.30.101

Configuring Syslog List of facilities separated by commas or *. Configured in /etc/syslog.conf Format: selector <Tab> action Ex: mail.info /var/log/mail.log Selector components Source (facility) List of facilities separated by commas or *. Importance (level) Can be none or *

/etc/syslog.conf # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log

Syslog Facilities Facility Used By kern The kernel user User processes (default) mail Mail servers and related software. daemon System daemons (except mail, cron) auth Security and authorization-related commands. lpr Print server and related commands. cron Cron daemon. local0-7 Eight local levels for other programs.

Syslog Levels Level Meaning emerg Panic situations (hardware failure, crash) alert Urgent situations crit Critical situations err Non-critical errors. warning Warnings. notice Might merit investigation. info Informational messages. debug Debugging (typically enabled temporarily.)

Syslog Actions Action Meaning filename Write message to file on local machine. @hostname Send message to syslogd on hostname. @ip Send message to syslogd at IP address. user1,user2 Write message to user screen if logged in. * Write message to all logged-in users.

Testing Syslog stu> for i in {debug,info,notice,warning,err,crit,alert,emerg} > do > logger -p daemon.$i "Test message for daemon, level $i" > done stu> tail /var/log/daemon.log Feb 11 15:57:00 localhost stu: Test message for daemon, level debug Feb 11 15:57:00 localhost stu: Test message for daemon, level info Feb 11 15:57:00 localhost stu: Test message for daemon, level notice Feb 11 15:57:00 localhost stu: Test message for daemon, level warning Feb 11 15:57:00 localhost stu: Test message for daemon, level err Feb 11 15:57:00 localhost stu: Test message for daemon, level crit Feb 11 15:57:00 localhost stu: Test message for daemon, level alert Feb 11 15:57:00 localhost stu: Test message for daemon, level emerg

Syslog messages are free form Syslog provides timestamp + hostname if log entries sent over network. Programmer must specify Facility Level (severity) But remaining data is free form. Different programs include different data and put same data in different locations. If you want to put it in a database, you have to parse the fields of interest out yourself for each data source.

Syslog Variants Some use m4 macros Red Hat Linux variants auth.notice ifdef(‘LOGHOST’, ‘/var/log/authlog’, ‘@loghost’) Red Hat Linux variants Allows spaces as separators. New operators: = (this priority only) Ex: mail.=info New operators: ! (except this pri and higher) Ex: mail.info,mail.!err

Syslog NG Free drop-in replacement for syslog. More configurable Save logs to templated location (auto-rotates.) Filter logs based on program, time, message, etc. Message format customization. Allows easy logging to remote database. Improved networking TCP support as well as UDP. Improved security Doesn’t trust hostnames in remote messages. TCP transmission permits encrypted tunneling (stunnel.)

Centralized Logging Collect all syslog data on one server. Allows logging to scale to large networks. Logs can be correlated across machines. Security-sensitive logs not on compromised host. Routers and diskless-hosts must log to a server. Need two syslog.conf files Client: sends all logs across network to server. Server: saves logs to database or local files.

Distributed Logging w/ Central Server Figure 1.3 from Logging and Log Management

Log Monitoring Too much data for a human to process. Logs arrive 24x7 too. Use a simple automatic monitoring program Triggers on patterns found in log. Examples: logwatch, swatch # 3ware logs watchfor /(?i)3w-xxxx.+no longer fault tolerant/ mail=root,subject=LW warn: disk 3ware RAID not fault tolerant throttle 1:00:00,use=regex

SIEM Security Information and Event Management (SIEM) Agents collect data from devices, servers, desktops, etc. SIEM aggregates information from multiple sources. Correlates data to establish relationships between log entries on different devices. ELK stack is most popular open source SIEM Elasticsearch stores and searches log data. Logstash collects log data from many sources. Kibana visualizes log information. OSSIM and Prelude are other popular OS SIEMs

ELK Screenshot

References Michael Bower, Building Secure Servers with Linux, O’Reilly, 2005. Anton A. Chuvakin and Kevin J. Schmidt. Logging and Log Management. Syngress. 2012. Evi Nemeth et al, UNIX System Administration Handbook, 3rd edition, Prentice Hall, 2001. NIST. SP 800-92: Guide to Computer Security Log Management. https://csrc.nist.gov/publications/detail/sp/800-92/final. 2006. Marcus Ranum. System Logging and Log Analysis. http://ranum.com/security/computer_security/archives/logging-notes.pdf

Released under CC BY-SA 3.0 This presentation is released under the Creative Commons Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0) license You are free: to Share — to copy and redistribute the material in any medium to Adapt— to remix, build, and transform upon the material to use part or all of this presentation in your own classes Under the following conditions: Attribution — You must attribute the work to James Walden, but cannot do so in a way that suggests that he endorses you or your use of these materials. Share Alike — If you remix, transform, or build upon this material, you must distribute the resulting work under this or a similar open license. Details and full text of the license can be found at https://creativecommons.org/licenses/by-nc-sa/3.0/