1. CIT 485: Advanced Cybersecurity
Logs

2. Topics System logs Finding logs Syslog and syslog-ng
Centralized logging
Log monitoring
SIEM

3. System Logs Logs record status and error conditions.
Where do log messages come from?
Kernel
Accounting system
System services
Application and server software
Logging methods:
Software records own logs (apache, cron).
UNIX software uses syslog service to manage logs.
Windows software uses OS Event Log service.
Network devices use syslog or SNMP traps.

4. Finding Logs On UNIX systems, most logs are stored under
/var/log
/var/adm
Check syslog's configuration
/etc/syslog.conf
To find other logs, read startup scripts
/etc/init.d/*
and manuals for services started by scripts.

5. Security Logs Fewer than 1% of OS log entries relate to security.
Similar percentages for most network devices, but
100% of firewall or IDS logs are relevant to security.
Important security log files on Linux include
auth.log, which records logins and su+sudo activity.
wtmp, which records additional login data.

6. Finding Logs Log file Program Contents messages syslog
Various program/kernel logs.
auth.log
su, ssh, login
Authorization fail/success.
lastlog
login, xdm
Logins, commands.
wtmp
login
Login accounting data.
acct/pacct
kernel
UNIX process accounting.
Xorg.log
X-Windows
X-Windows failures/info.

7. Syslog Comprehensive logging system. Sorts messages by
Frees programmers from managing log files.
Gives sysadmins control over log management.
Sorts messages by
Sources
Importance
Routes messages to destinations
Files
Network
Terminals

8. Syslog Components Syslog openlog, syslog, closelog logger
Daemon that does actual logging.
Additional daemon, klog, gets kernel messages.
openlog, syslog, closelog
C library routines to submit logs to syslog.
logger
User-level program to submit logs to syslog.
Can use from shell scripts.

9. Example Syslog Messages
Feb 11 10:17:01 localhost /USR/SBIN/CRON[1971]: (root) CMD ( run-parts --report /etc/cron.hourly)
Feb 11 10:37:22 localhost -- MARK --
Feb 11 10:51:11 localhost dhclient: DHCPREQUEST on eth1 to port 67
Feb 11 10:51:11 localhost dhclient: DHCPACK from
Feb 11 10:51:11 localhost dhclient: bound to renewal in seconds.
Feb 11 14:37:22 localhost -- MARK --
Feb 11 14:44:21 localhost mysqld[7340]: :44:21 /usr/sbin/mysqld: Normal shutdown
Feb 12 04:46:42 localhost sshd[29093]: Address maps to ns.thundernet.co.kr, but this does not map back to the address - POSSIBLE BREAKIN ATTEMPT!
Feb 12 04:46:44 localhost sshd[29097]: Invalid user matt from ::ffff:

10. Configuring Syslog List of facilities separated by commas or *.
Configured in /etc/syslog.conf
Format: selector <Tab> action
Ex: mail.info /var/log/mail.log
Selector components
Source (facility)
List of facilities separated by commas or *.
Importance (level)
Can be none or *

11. /etc/syslog.conf # Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log

12. Syslog Facilities Facility Used By kern The kernel user
User processes (default)
mail
Mail servers and related software.
daemon
System daemons (except mail, cron)
auth
Security and authorization-related commands.
lpr
Print server and related commands.
cron
Cron daemon.
local0-7
Eight local levels for other programs.

13. Syslog Levels Level Meaning emerg
Panic situations (hardware failure, crash)
alert
Urgent situations
crit
Critical situations
err
Non-critical errors.
warning
Warnings.
notice
Might merit investigation.
info
Informational messages.
debug
Debugging (typically enabled temporarily.)

14. Syslog Actions Action Meaning filename
Write message to file on local machine.
@hostname
Send message to syslogd on hostname.
@ip
Send message to syslogd at IP address.
user1,user2
Write message to user screen if logged in. *
Write message to all logged-in users.

15. Testing Syslog
stu> for i in {debug,info,notice,warning,err,crit,alert,emerg}
> do
> logger -p daemon.$i "Test message for daemon, level $i"
> done
stu> tail /var/log/daemon.log
Feb 11 15:57:00 localhost stu: Test message for daemon, level debug
Feb 11 15:57:00 localhost stu: Test message for daemon, level info
Feb 11 15:57:00 localhost stu: Test message for daemon, level notice
Feb 11 15:57:00 localhost stu: Test message for daemon, level warning
Feb 11 15:57:00 localhost stu: Test message for daemon, level err
Feb 11 15:57:00 localhost stu: Test message for daemon, level crit
Feb 11 15:57:00 localhost stu: Test message for daemon, level alert
Feb 11 15:57:00 localhost stu: Test message for daemon, level emerg

16. Syslog messages are free form
Syslog provides timestamp
+ hostname if log entries sent over network.
Programmer must specify
Facility
Level (severity)
But remaining data is free form.
Different programs include different data and put same data in different locations.
If you want to put it in a database, you have to parse the fields of interest out yourself for each data source.

17. Syslog Variants Some use m4 macros Red Hat Linux variants
auth.notice ifdef(‘LOGHOST’, ‘/var/log/authlog’,
Red Hat Linux variants
Allows spaces as separators.
New operators: = (this priority only)
Ex: mail.=info
New operators: ! (except this pri and higher)
Ex: mail.info,mail.!err

18. Syslog NG Free drop-in replacement for syslog. More configurable
Save logs to templated location (auto-rotates.)
Filter logs based on program, time, message, etc.
Message format customization.
Allows easy logging to remote database.
Improved networking
TCP support as well as UDP.
Improved security
Doesn’t trust hostnames in remote messages.
TCP transmission permits encrypted tunneling (stunnel.)

19. Centralized Logging Collect all syslog data on one server.
Allows logging to scale to large networks.
Logs can be correlated across machines.
Security-sensitive logs not on compromised host.
Routers and diskless-hosts must log to a server.
Need two syslog.conf files
Client: sends all logs across network to server.
Server: saves logs to database or local files.

20. Distributed Logging w/ Central Server
Figure 1.3 from Logging and Log Management

21. Log Monitoring Too much data for a human to process.
Logs arrive 24x7 too.
Use a simple automatic monitoring program
Triggers on patterns found in log.
Examples: logwatch, swatch
# 3ware logs watchfor /(?i)3w-xxxx.+no longer fault tolerant/ mail=root,subject=LW warn: disk 3ware RAID not fault tolerant throttle 1:00:00,use=regex

22. SIEM Security Information and Event Management (SIEM)
Agents collect data from devices, servers, desktops, etc.
SIEM aggregates information from multiple sources.
Correlates data to establish relationships between log entries on different devices.
ELK stack is most popular open source SIEM
Elasticsearch stores and searches log data.
Logstash collects log data from many sources.
Kibana visualizes log information.
OSSIM and Prelude are other popular OS SIEMs

23. ELK Screenshot

24. References
Michael Bower, Building Secure Servers with Linux, O’Reilly, 2005.
Anton A. Chuvakin and Kevin J. Schmidt. Logging and Log Management. Syngress
Evi Nemeth et al, UNIX System Administration Handbook, 3rd edition, Prentice Hall, 2001.
NIST. SP : Guide to Computer Security Log Management
Marcus Ranum. System Logging and Log Analysis.

25. Released under CC BY-SA 3.0
This presentation is released under the Creative Commons Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0) license
You are free:
to Share — to copy and redistribute the material in any medium
to Adapt— to remix, build, and transform upon the material
to use part or all of this presentation in your own classes
Under the following conditions:
Attribution — You must attribute the work to James Walden, but cannot do so in a way that suggests that he endorses you or your use of these materials.
Share Alike — If you remix, transform, or build upon this material, you must distribute the resulting work under this or a similar open license.
Details and full text of the license can be found at