1. Syslog and Log Files
Chapter 11

2. Chapter 11 - Syslog and Log Files
Introduction
The accounting system, the kernel, and carious utilities emit data that is logged and eventually ends up on your finite-sized disks.
Most of the data has a limited useful lifetime and needs to be summarized, compressed, archived, and eventually thrown away.
Chapter 11 - Syslog and Log Files

3. Chapter 11 - Syslog and Log Files
1. Logging Policies
Logging policies vary from site to site.
Common schemes include:
Throw it all away immediately
Reset log files at periodic intervals
Rotate log files, keeping data for a fixed time.
Compress and archive logs to tape or other permanent media.
Whatever scheme you select, maintenance of log files should be automated with cron (Chapter 9)
Chapter 11 - Syslog and Log Files

4. Chapter 11 - Syslog and Log Files
1. Logging Policies
Throwing away log files
We do not recommend throwing away all logging information.
Log files provide important evidence of break ins and are helpful for alerting you to hardware and software problems.
In general, given enough disk space, data should be kept for at least a moth and then discarded.
If you need further back than that, you can go to backups.
Chapter 11 - Syslog and Log Files

5. Chapter 11 - Syslog and Log Files
1. Logging Policies
Rotating log files
Most sites store each day’s log information on disk
Sometimes in a compressed format.
Keeping them online allows them to be searched (with grep)
Dedicating a disk partition to logs is also an intelligent decision.
Chapter 11 - Syslog and Log Files

6. Chapter 11 - Syslog and Log Files
1. Logging Policies
One common way to limit disk usage is to rotate log files.
If a log file is called logfile, for example, the backup copies might be called logfile.1, logfile.2, and so on.
You could use a shell script to implement this rotation
#!/bin/sh
cd /var/log
mv logfile.2 logfile.3
mv logfile.1 logfile.2
mv logfile logfile.1
cat /dev/null > logfile
chmod 600 logfile
Ownership is important, so you may need a chown added.
Chapter 11 - Syslog and Log Files

7. Chapter 11 - Syslog and Log Files
1. Logging Policies
Many systems supply an off-the-shelf log rotation script that is run out of cron.
By all means, use the standard script if it does what you want.
If your system doesn’t supply a rotation system, we suggest that you use a Perl script called rotz written by Matt Segur and Michael Bernstein for this purpose.
It’s available from
Chapter 11 - Syslog and Log Files

8. Chapter 11 - Syslog and Log Files
1. Logging Policies
Archiving log files
Some sites must archive all accounting data and log files as a matter of policy, perhaps to provide data for a potential audit.
Log files should always be included in your regular backup sequence.
They may also be archived to a separate tape series.
Chapter 11 - Syslog and Log Files

9. Chapter 11 - Syslog and Log Files
2. Finding Log Files
UNIX is often criticized for being inconsistent, and indeed it is.
Just take a look at log files and you’re sure to find some with names like maillog, some like ftp.log, and maybe even some like lpNet, lpd-errs, or console-log.
In addition to having random names, some are often scattered across directories and filesystems.
This section attempts to help you find all the files that are quietly taking over your disk
And suggest a granularity for dealing with each.
Chapter 11 - Syslog and Log Files

10. Chapter 11 - Syslog and Log Files
2. Finding Log Files
To locate your log files, read your system’s startup scripts (/etc/rc*, /etc/rc.d/*, or /etc/init.d/*) to see if logging is turned on when daemons are run.
Table 11.1 compiles information about some of the more common log files on our example systems.
Filenames are relative to /var/adm or /var/log unless otherwise noted.
Log files usually have mode 644
some sites reduce permissions to 640 or 600
sulog, sudo.log and others should be 600
Chapter 11 - Syslog and Log Files

11. Chapter 11 - Syslog and Log Files
2. Finding Log Files
Chapter 11 - Syslog and Log Files

12. Chapter 11 - Syslog and Log Files
3. Files NOT to Manage
You might be tempted to manage all log files with a rotation and archiving scheme.
But there are two files that you should not touch
/var/adm/lastlog
records each user’s last login. It is a sparse file indexed by UID (and it has holes)
/etc/utmp
utmp attempts to keep a record of each user that is currently logged in
Chapter 11 - Syslog and Log Files

13. Chapter 11 - Syslog and Log Files
4. Vendor Specifics
Vendors seem to have hidden log files all over the disk.
Careful detective work with your daemon’s config files and your syslog configuration file will find many of them.
This section details some of the more obscure nooks and crannies in which log files have been hidden.
Chapter 11 - Syslog and Log Files

14. Chapter 11 - Syslog and Log Files
4. Vendor Specifics
Solaris has the most disorganized collection of log files ever.
With a directory called /var/log it shouldn’t be so hard.
Here are a few places to look
/var/log/*
/var/cron/log
/var/lp/logs/*
/var/saf/_log
/var/saf/zsmon/log
/var/adm/{messages, aculog, sulog, vold.log, wtmpx}
Chapter 11 - Syslog and Log Files

15. Chapter 11 - Syslog and Log Files
4. Vendor Specifics
HP-UX log files are in /var/adm
There are a lot of odd little mystery files in this directory, many of which are not log files,
so be careful what you touch
By default, all log entries submitted via syslog go into the /var/adm/syslog directory.
Chapter 11 - Syslog and Log Files

16. Chapter 11 - Syslog and Log Files
4. Vendor Specifics
Red Hat gets a gold star for logging sanity.
Not only are logs clearly named and consistently stored in /var/log,
but Red Hat also provides a superior tool, logrotate, for rotating, truncating, and managing them
new software packages can drop a configuration file into /etc/logrotate.d directory to set up a management strategy for their log files.
Chapter 11 - Syslog and Log Files

17. Chapter 11 - Syslog and Log Files
4. Vendor Specifics
FreeBSD is another prize winner in the realm of logging.
Log files are generally found in /var/log,
although cron’s log is kept in /var/cron
and accounting files are kept in /var/account
The newsyslog utility is responsible for managing and rotating logs.
It runs out of cron and takes its marching orders from /etc/newsyslog.conf
Chapter 11 - Syslog and Log Files

18. 5. syslog: The System Event Logger
syslog, originally written by Eric Allman, is a comprehensive logging system.
Many vendors use syslog to manage the information generated by the kernel and the system utilities.
Syslog has two important functions:
to liberate programmer from the tedious mechanics of writing log files
and to put administrators in control of logging
Chapter 11 - Syslog and Log Files

19. 5. syslog: The System Event Logger
Syslog is quite flexible.
It allows messages to be sorted by their source and importance
and routed to a variety of destinations
log files,
user terminals
other machines
Chapter 11 - Syslog and Log Files

20. 5. syslog: The System Event Logger
syslog consists of three parts:
syslogd
the logging daemon (config file is /etc/syslog-conf)
openlog (et al.)
library routines that submit messages to syslogd
logger
a user-level command that submits entries from the shell
Chapter 11 - Syslog and Log Files

21. 5. syslog: The System Event Logger
syslogd is started at boot time
programs that are syslog aware write log entries to a special file (sometimes called /dev/log) which is either a UNIX domain socket, a named pipe, or a STREAM module (depending on the system)
syslogd reads the messages, consults its configuration files, and dispatches each message to the appropriate destinations.
Chapter 11 - Syslog and Log Files

22. 5. Syslog: The System Event Logger
Configuring syslogd
The configuration file /etc/syslog.conf controls syslogd’s behavior.
It is a text file with a relatively simple format.
Selector <Tab> action
for example:
mail.info /var/log/maillog
Selectors are of the format facility.level
Chapter 11 - Syslog and Log Files

23. 5. Syslog: The System Event Logger
Chapter 11 - Syslog and Log Files

24. 5. Syslog: The System Event Logger
Chapter 11 - Syslog and Log Files

25. 5. Syslog: The System Event Logger
Chapter 11 - Syslog and Log Files

26. 5. Syslog: The System Event Logger
Chapter 11 - Syslog and Log Files

27. 6. Condensing Log Files to useful information
Syslog is great for sorting and routing log messages, but when all is said and done, the end product is still a bunch of log files.
While they may contain all kinds of useful information, those files aren’t going to come and find you when something goes wrong.
Another layer of software is needed to analyze the logs and make sure some important messages don’t get lost amid the chatter.
Chapter 11 - Syslog and Log Files

28. 6. Condensing Log Files to useful information
A variety of free tools are available to fill this niche, and most of them are pretty similar:
They scan recent log entries,
match them against a database of regular expressions,
and process the important messages in some attention-getting way
Chapter 11 - Syslog and Log Files

29. 6. Condensing Log Files to useful information
Tools differ primarily in their degree of flexibility and in the size of their off-the-shelf database of patterns.
Two of the more commonly used log postprocessors are
swatch - perl script -
logcheck - sh script -
Chapter 11 - Syslog and Log Files

30. 6. Condensing Log Files to useful information
No matter what system you use to scan files, there are a couple of things you should be sure to look for
Most security related messages
monitor failed login attempts, su, and sudo attempts
Messages about disks that have filled up
Full disks often bring all useful work to a standstill
Messages that are repeated many times
If only to clean things up
Chapter 11 - Syslog and Log Files

31. Chapter 11 - Syslog and Log Files