Linux System Log Rotation

/etc/logrotate.d

--------------------------------------------------------------------------------

Setting up and verifying system logging: syslog and klog; /etc/syslog.conf;
remote logging; monitoring logs using swatch; managing logs using log rotate.

It is possible to force log rotation to test new configuration.

{
dbw comment: invalid entries in logrotate.conf - such as logs for customer
web directories since deleted, can cause logrotate to fail completely! You
_can_ force a log rotation checking your logrotate.conf file for validity.
syntax: logrotate -f /etc/logrotate.conf

Test without actually rotating and show details: logrotate -vd /etc/logrotate.conf

}

man logrotate

--------------------------------------------------------------------------------
#note: endscript should appear only in the last entry if in logrotate.d
# and should not appear at all if in logrotate.conf
--------------------------------------------------------------------------------
How logrotate is invoked from cron.
On one of my RedHat 5.2 systems, `crond' is started from the script
/etc/rc.d/init.d/crond. The manual `man cron' says that cron

searches for /etc/crontab and the files in the /etc/cron.d/ directory.

My /etc/crontab file contains the following.
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
0 0 1 * * root run-parts /etc/cron.monthly

And the /etc/cron.d directory contains nothing.
So clearly, any logrotate commands are being run out of one or more of the
/etc/cron.* files. The manual `man 5 crontab' indicates that the above
commands mean that the program `run-parts' will be run as user `root' for
each of the scripts.

The command `run-parts' turns out to be the script /usr/bin/run-parts, which
is very short, as follows.

#!/bin/bash

# run-parts - concept taken from Debian

# keep going when something fails
set +e

if [ $# -lt 1 ]; then
echo "Usage: run-parts <dir>"
exit 1
fi

if [ ! -d $1 ]; then
echo "Not a directory: $1"
exit 1
fi

for i in $1/* ; do
[ -d $i ] && continue
if [ -x $i ]; then
$i
fi
done

exit 0

In essence, this just runs all of the scripts in the specified directory.
E.g. each hour, the executable plain files in /etc/cron.hourly are run.
It turns out that the `logrotate' program is invoked from /etc/cron.daily.
This is the contents of the file /etc/cron.daily/logrotate.

#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf

This means that the only file that `logrotate' gets its instructions from
directly is /etc/logrotate.conf, which contains the following lines.
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 8 weeks worth of backlogs
rotate 8

# send errors to root
errors root

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own lastlog or wtmp -- we'll rotate them here
/var/log/wtmp {
monthly
rotate 1
}

# system-specific logs may be configured here

This is all explained in the manual `man logrotate'.

Operation of logrotate.
The directory /etc/logrotate.d contains the following files.

-rw-r--r-- 1 root root 354 Oct 13 1998 apache
-rw-r--r-- 1 root root 108 Aug 28 1999 cron
-rw-r--r-- 1 root root 188 Oct 14 1998 linuxconf
-rw-r--r-- 1 root root 156 Oct 13 1998 mgetty
-rw-r--r-- 1 root root 327 Aug 12 1998 syslog
-rw-r--r-- 1 root root 457 Sep 10 1998 uucp

The file /etc/logrotate.d/apache is the one I'm interested in for this
exercise. This file contains the following.

/var/log/httpd/access_log {
postrotate
/usr/bin/killall -HUP httpd
endscript
}

/var/log/httpd/agent_log {
postrotate
/usr/bin/killall -HUP httpd
endscript
}

/var/log/httpd/error_log {
postrotate
/usr/bin/killall -HUP httpd
endscript
}

/var/log/httpd/referer_log {
postrotate
/usr/bin/killall -HUP httpd
endscript
}

When I installed the latest version of Apache to get the PHP3 and
PostgreSQL to work (around 20 March 2000) on my web server machine,
I installed Apache so that the log files were in /home2/apache/logs
instead of /var/log/httpd.

Therefore what I need to do now is to modify the
/etc/logrotate.d/apache file so that the files referred to are all
in the directory /home2/apache/logs instead. My new
/etc/logrotate.d/apache script is as follows, and I saved the old
one in directory /etc/logrotate.d/old1.

# The new improved logrotate script for apache on fox.

/home2/apache/logs/*-access_log {
rotate 9
monthly
errors akenning@fox.topology.org
create
ifempty
olddir /home2/apache/logs/oldlogs
postrotate
/usr/bin/killall -HUP httpd
endscript
}

/home2/apache/logs/*-combref_log {
rotate 9
monthly
errors akenning@fox.topology.org
create
ifempty
olddir /home2/apache/logs/oldlogs
postrotate
/usr/bin/killall -HUP httpd
endscript
}

/home2/apache/logs/*-error_log {
rotate 9
monthly
errors akenning@fox.topology.org
create
ifempty
olddir /home2/apache/logs/oldlogs
postrotate
/usr/bin/killall -HUP httpd
endscript
}

--------------------------------------------------------------------------------

Now on Sunday 30 September 2001, it's time to get `logrotate' going on my new
SuSE 7.1 web server `dog'. The problem is that SuSE 7.1 does not come with
logrotate software on it. So I had to go and look for it. The link at redhat
was
wrong, but by looking around a bit, I finally found it at
ftp://ftp.redhat.com/pub/redhat/linux/code/logrotate/
I downloaded this file: logrotate-3.3.tar.gz. This can be compiled very simply
with `make' and `make install'. There's a manual logrotate.8 with it too. The
binary is installed as /usr/sbin/logrotate. So now all I have to do is write a
configuration file for the logrotate program and then write a cron script for
it.
(Note that since my main initial consideration is to rotate the httpd logs for
the beginning of October 2001, I could just use the Apache tool for this
purpose,
which is /usr/local/apache/bin/rotatelogs on my system. However, I can't
understand the documentation for this. So I'm playing safe and using the more
flexible redhat `logrotate' tool instead.)

On my SuSE 7.1 machine `dog', the manual says that after reading the per-user
crontab files in /var/spool/cron/tabs on start-up, the cron process reads the
file /etc/crontab. On my machine as configured, I find the following.

root@dog# more /etc/crontab
SHELL=/bin/sh
PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin
MAILTO=root

#-* * * * * root test -x /usr/sbin/atrun && /usr/sbin/atrun
0 21 * * * root test -x /usr/sbin/faxqclean && /usr/sbin/faxqclean
5 22 * * * root test -x /usr/sbin/texpire && /usr/sbin/texpire
25 23 * * * root test -e /usr/sbin/faxcron && sh /usr/sbin/faxcron | mail
FaxMaster

#
# check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly
#
-*/15 * * * * root test -x /usr/lib/cron/run-crons &&
/usr/lib/cron/run-crons
59 * * * * root rm -f /var/spool/cron/lastrun/cron.hourly
14 0 * * * root rm -f /var/spool/cron/lastrun/cron.daily
29 0 * * 6 root rm -f /var/spool/cron/lastrun/cron.weekly
44 0 1 * * root rm -f /var/spool/cron/lastrun/cron.monthly

There's nothing here to help me with initiating my daily script. (By the way,
the fax commands are a bit worrying. I'll get rid of those when I understand
exactly what they do. They obviously produce many meaningless message which
root receives every day!)

So my next step is to look at the files in /etc/cron.d, because the cron
manual
says that all scripts in this directory are read next. On my machine the only
file in /etc/cron.d is a script `seccheck', which produces copious useless
messages to root every day and week. (I'll see if I can get rid of that some
day too!)

The directory /etc/cron.daily contains a script `aaa_base_rotate_logs' which
contains a complex set of rotation rules, but how are the scripts in this
directory invoked? Hmmm... Maybe they're invoked from that
`/usr/lib/cron/run-crons' script. Yes!! That's where it's invoked from. Yet
another big, incomprehensible script. The core of that script is the following
Bourne-shell loop.

SPOOL=/var/spool/cron/lastrun
for CRONDIR in /etc/cron.{hourly,daily,weekly,monthly} ; do
test -d $CRONDIR || continue
BASE=${CRONDIR##*/}
test -e $SPOOL/$BASE && {
case $BASE in
cron.hourly) TIME="-cmin +60 -or -cmin 60" ;;
cron.daily) TIME="-ctime +1 -or -ctime 1" ;;
cron.weekly) TIME="-ctime +7 -or -ctime 7" ;;
cron.monthly) TIME="-ctime +`date -u +%d`" ;;
esac
eval find $SPOOL/$BASE $TIME | xargs -r rm -f
}
if test ! -e $SPOOL/$BASE ; then
touch $SPOOL/$BASE

# keep going when something fails
set +e
for SCRIPT in $CRONDIR/*[^~,] ; do
test -d $SCRIPT && continue
test -x $SCRIPT || continue
case "$SCRIPT" in
*.rpm*) continue ;;
*.swap) continue ;;
*.bak) continue ;;
*.orig) continue ;;
\#*) continue ;;
esac
/sbin/checkproc $SCRIPT && continue
nice -15 $SCRIPT
done
fi
done

Now what does this mean?
The command `BASE=${CRONDIR##*/}' means that BASE is set to the `longest
substring of $CRONDIR which matches pattern "*/"', according to my Bash
reference card. This just means that the leading path components are removed.
(This is done more simply in C-shell!) In the case of the daily cron job, if
there is a file /var/spool/cron/lastrun/cron.daily (which is true), then the
following command is run.
eval find /var/spool/cron/lastrun/cron.daily -ctime +1 -or -ctime 1 | xargs -r
rm -f

The `xargs' command (which I have never seen before) builds and executes a
command line from standard input. Wierd! The `xargs' manual says this.

If the standard input does not contain any non-blanks,
do not run the command. Normally, the command is run once
even if there is no input.

So in this case, the command `rm -f' is executed for each of the files with
modification times within 24 hours of the current time. I don't really
understand this.

It looks like the file removal commands in the file /etc/crontab are designed
to synchronise the operation of the quarter-hour operations. The commands are
only executed if the files in /var/spool/cron/lastrun have been removed. What
a convoluted way of achieving a simple objective!!

Next any file in the directory /etc/cron.daily which do not end in the
characters `~' or `,' (presumed to be edited files) are executed if they are
executable and do not have the endings .rpm, .swap, .bak or .orig as follows.

/sbin/checkproc $SCRIPT && continue
nice -15 $SCRIPT

The loop continues is the process is already running. Otherwise it is run with
nice level 15.

What this all means finally is that any script in the directory
/etc/cron.daily
will be run at 00:00 on each day. More to the point, the /etc/cron.monthly
scripts are run at 00:00 at the beginning of each month. It all looks a bit
dodgy because the `date' test is `date -u', which gives the current UTC day.
But this should work, although the motivation for the `-u' is not quite clear.

All I have to do now is write a script and put it in /etc/cron.monthly. No
problem!

--------------------------------------------------------------------------------
sample methods:

# system-specific logs may be configured here
## rotate apache logs weekly, keeping 8 rotations
/www/apache/var/logs/*log {
rotate 8
postrotate
kill -HUP `cat /www/apache/var/logs/httpd.pid`
endscript
}
## rotate rembo logs weekly, keeping 4 rotations
/opt/rembo/logs/*.log {
rotate 4
postrotate
/etc/rc.d/init.d/rembo stop
/etc/rc.d/init.d/rembo start
endscript
}

--------------------------------------------------------------------------------
the roach method (for virtual hosting):

I have many virtual hosts. I feared that apache would be start/stop
frequently,
once per virtual host, during the rotation process. Therefore, I decided to
use
the 'copytruncate' option. With this option the apache web server does not
require a restart. -roach

/usr/apache/log/*log {
weekly
rotate 52
notifempty
missingok
copytruncate
}

/usr/apache/log/dynahost.dbw.org/*log {
weekly
rotate 52
notifempty
missingok
copytruncate
}

--------------------------------------------------------------------------------
Configure the new /etc/logrotate.d/apache file
Now Apache logs files residing in the /chroot/var/log/httpd directory instead
of
/var/log/httpd and for this reason we need to modify the
/etc/logrotate.d/httpd
file to point to the new chrooted directory. Also, we've compiled Apache with
mod_ssl, so we'll add one more line to permit the logrotate program to rotate
the ssl_request_log and ssl_engine_log files. Configure your
/etc/logrotate.d/apache file to rotate your log files each week automatically.

Create the apache file, touch /etc/logrotate.d/apache and add:
/chroot/httpd/var/log/httpd/access_log {
missingok
postrotate
/usr/bin/killall -HUP /chroot/httpd/usr/sbin/httpd
endscript
}

/chroot/httpd/var/log/httpd/error_log {
missingok
postrotate
/usr/bin/killall -HUP /chroot/httpd/usr/sbin/httpd
endscript
}

/chroot/httpd/var/log/httpd/ssl_request_log {
missingok
postrotate
/usr/bin/killall -HUP /chroot/httpd/usr/sbin/httpd
endscript
}

/chroot/httpd/var/log/httpd/ssl_engine_log {
missingok
postrotate
/usr/bin/killall -HUP /chroot/httpd/usr/sbin/httpd
endscript
}

--------------------------------------------------------------------------------
Credits and Quasi-bibliography:

The majority of this text is from:
http://www.topology.org/linux/logrotate.html
Reference: Alan Kennington's

Some information was gathered from:
http://misc.epfl.ch/rembo/logrotate_conf.html
Reference: Unknown

Securing and Optimizing Linux: RedHat Edition -A Hands on Guide
http://www.tldp.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/chap29sec258.html

Anything else was created by:
roach

date: Sun May 12 18:03:33 CDT 2002
--------------------------------------------------------------------------------

Logrotate: error reading top line of /var/lib/logrotate.status

If you get an error saying

/etc/cron.daily/logrotate:

error: error reading top line of /var/lib/logrotate.status

delete /var/lib/logrotate.status, and then run logrotate once with -f flag,
like

#> logrotate -f /etc/logrorate.d/syslog

This should initialize the status file and the error should not be repeating.

Virtual Host Log Rotation for Apache

Many linux distributions include a utility called logrotate.d which is installed and configured by default. It is configured to rotate the main Apache log file typically weekly or monthly. Yet, Apache configuration for virtual hosts (virtual_host) suggests creating separate logs for each virtual host, and in a directory named for the site. This is typically under the primary Apache log directory. Consider a server where /var/log/apache2/access.log is the primary apache log, but there is a virtual host which a log under /var/log/apache2/www.smokinghotbabes.com/access.log. Now looking in /etc/logrotate.d/httpd the default config is:

/var/log/apache2/*.log {
    missingok
    notifempty
    postrotate 
	/usr/bin/killall -HUP httpd
    endscript
}

This default installed configuration for logrotate.d/httpd will only rotate the primary web site. It will not rotate virtual host sites. This can be amended by adding another part to the log rotation configuration script:

/var/log/apache2/*.log {
    missingok
    notifempty
    postrotate 
	/usr/bin/killall -HUP httpd
    endscript
}

/var/log/apache2/*/*.log {
    missingok
    notifempty
    postrotate 
	/usr/bin/killall -HUP httpd
    endscript
}

configuration testing and some important flags

Verbose

-v tells logrotate to say what it’s doing while it’s doing it. It’s very useful when trying to find out why logrotate doesn’t rotate a log when you want it to.

Debug

-d tells logrotate to go through the motions of rotating logs but not actually rotate them. Good for checking that the config file is formatted properly and that logrotate can find the log files it would rotate. It doesn’t actually run the rotations and it doesn’t test some parts of the process like the postrotate scripts.

Force

-f forces logrotate to rotate all logs, whether or not they would normally need to be rotated at that time.

External Resources

Linux System Log Rotation

Contents

Virtual Host Log Rotation for Apache

configuration testing and some important flags

Verbose

Debug

Force

External Resources

Navigation menu

Linux System Log Rotation

Virtual Host Log Rotation for Apache

configuration testing and some important flags

Verbose

Debug

Force

External Resources

Navigation menu

Search