Difference between revisions of "Linux System Log Rotation"

Revision as of 18:19, 26 April 2013

/etc/logrotate.d

--------------------------------------------------------------------------------

Setting up and verifying system logging: syslog and klog; /etc/syslog.conf;
remote logging; monitoring logs using swatch; managing logs using log rotate. 

It is possible to force log rotation to test new configuration.

{
  dbw comment: invalid entries in logrotate.conf - such as logs for customer 
  web directories since deleted, can cause logrotate to fail completely!  You
  _can_ force a log rotation checking your logrotate.conf file for validity.
  syntax: logrotate -f /etc/logrotate.conf
}

man logrotate

--------------------------------------------------------------------------------
#note: endscript should appear only in the last entry if in logrotate.d
#      and should not appear at all if in logrotate.conf
--------------------------------------------------------------------------------
How logrotate is invoked from cron. 
On one of my RedHat 5.2 systems, `crond' is started from the script 
/etc/rc.d/init.d/crond. The manual `man cron' says that cron 

searches for /etc/crontab and the files in the /etc/cron.d/ directory.

My /etc/crontab file contains the following. 
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
0  0 1 * * root run-parts /etc/cron.monthly

And the /etc/cron.d directory contains nothing. 
So clearly, any logrotate commands are being run out of one or more of the 
/etc/cron.* files. The manual `man 5 crontab' indicates that the above 
commands mean that the program `run-parts' will be run as user `root' for 
each of the scripts. 

The command `run-parts' turns out to be the script /usr/bin/run-parts, which 
is very short, as follows. 

#!/bin/bash

# run-parts - concept taken from Debian

# keep going when something fails
set +e

if [ $# -lt 1 ]; then
        echo "Usage: run-parts <dir>"
        exit 1
fi

if [ ! -d $1 ]; then
        echo "Not a directory: $1"
        exit 1
fi

for i in $1/* ; do
        [ -d $i ] && continue
        if [ -x $i ]; then
                $i
        fi
done

exit 0

In essence, this just runs all of the scripts in the specified directory. 
E.g. each hour, the executable plain files in /etc/cron.hourly are run. 
It turns out that the `logrotate' program is invoked from /etc/cron.daily. 
This is the contents of the file /etc/cron.daily/logrotate. 

#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf

This means that the only file that `logrotate' gets its instructions from 
directly is /etc/logrotate.conf, which contains the following lines. 
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 8 weeks worth of backlogs
rotate 8

# send errors to root
errors root

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own lastlog or wtmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    rotate 1
}

# system-specific logs may be configured here

This is all explained in the manual `man logrotate'.  


Operation of logrotate. 
The directory /etc/logrotate.d contains the following files. 

-rw-r--r--   1 root     root          354 Oct 13  1998 apache
-rw-r--r--   1 root     root          108 Aug 28  1999 cron
-rw-r--r--   1 root     root          188 Oct 14  1998 linuxconf
-rw-r--r--   1 root     root          156 Oct 13  1998 mgetty
-rw-r--r--   1 root     root          327 Aug 12  1998 syslog
-rw-r--r--   1 root     root          457 Sep 10  1998 uucp

The file /etc/logrotate.d/apache is the one I'm interested in for this 
exercise. This file contains the following. 

/var/log/httpd/access_log {
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
}

/var/log/httpd/agent_log {
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
}

/var/log/httpd/error_log {
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
}

/var/log/httpd/referer_log {
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
}

When I installed the latest version of Apache to get the PHP3 and 
PostgreSQL to work (around 20 March 2000) on my web server machine, 
I installed Apache so that the log files were in /home2/apache/logs 
instead of /var/log/httpd. 

Therefore what I need to do now is to modify the 
/etc/logrotate.d/apache file so that the files referred to are all 
in the directory /home2/apache/logs instead. My new 
/etc/logrotate.d/apache script is as follows, and I saved the old 
one in directory /etc/logrotate.d/old1. 

# The new improved logrotate script for apache on fox.

/home2/apache/logs/*-access_log {
    rotate 9
    monthly
    errors akenning@fox.topology.org
    create
    ifempty
    olddir /home2/apache/logs/oldlogs
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
    }

/home2/apache/logs/*-combref_log {
    rotate 9
    monthly
    errors akenning@fox.topology.org
    create
    ifempty
    olddir /home2/apache/logs/oldlogs
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
    }

/home2/apache/logs/*-error_log {
    rotate 9
    monthly
    errors akenning@fox.topology.org
    create
    ifempty
    olddir /home2/apache/logs/oldlogs
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
    }

--------------------------------------------------------------------------------

Now on Sunday 30 September 2001, it's time to get `logrotate' going on my new 
SuSE 7.1 web server `dog'. The problem is that SuSE 7.1 does not come with 
logrotate software on it. So I had to go and look for it. The link at redhat
was 
wrong, but by looking around a bit, I finally found it at 
ftp://ftp.redhat.com/pub/redhat/linux/code/logrotate/ 
I downloaded this file: logrotate-3.3.tar.gz. This can be compiled very simply 
with `make' and `make install'. There's a manual logrotate.8 with it too. The 
binary is installed as /usr/sbin/logrotate. So now all I have to do is write a 
configuration file for the logrotate program and then write a cron script for
it. 
(Note that since my main initial consideration is to rotate the httpd logs for 
the beginning of October 2001, I could just use the Apache tool for this
purpose, 
which is /usr/local/apache/bin/rotatelogs on my system. However, I can't 
understand the documentation for this. So I'm playing safe and using the more 
flexible redhat `logrotate' tool instead.) 

On my SuSE 7.1 machine `dog', the manual says that after reading the per-user 
crontab files in /var/spool/cron/tabs on start-up, the cron process reads the 
file /etc/crontab. On my machine as configured, I find the following. 

root@dog# more /etc/crontab
SHELL=/bin/sh
PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin
MAILTO=root

#-* * * * *    root  test -x /usr/sbin/atrun && /usr/sbin/atrun
0 21 * * *     root  test -x /usr/sbin/faxqclean && /usr/sbin/faxqclean
5 22 * * *     root  test -x /usr/sbin/texpire && /usr/sbin/texpire
25 23 * * *    root  test -e /usr/sbin/faxcron && sh /usr/sbin/faxcron | mail
FaxMaster

#
# check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly
#
-*/15 * * * *   root  test -x /usr/lib/cron/run-crons &&
/usr/lib/cron/run-crons
59 *  * * *     root  rm -f /var/spool/cron/lastrun/cron.hourly
14 0  * * *     root  rm -f /var/spool/cron/lastrun/cron.daily
29 0  * * 6     root  rm -f /var/spool/cron/lastrun/cron.weekly
44 0  1 * *     root  rm -f /var/spool/cron/lastrun/cron.monthly

There's nothing here to help me with initiating my daily script. (By the way, 
the fax commands are a bit worrying. I'll get rid of those when I understand 
exactly what they do. They obviously produce many meaningless message which 
root receives every day!) 

So my next step is to look at the files in /etc/cron.d, because the cron
manual 
says that all scripts in this directory are read next. On my machine the only 
file in /etc/cron.d is a script `seccheck', which produces copious useless 
messages to root every day and week. (I'll see if I can get rid of that some 
day too!) 

The directory /etc/cron.daily contains a script `aaa_base_rotate_logs' which 
contains a complex set of rotation rules, but how are the scripts in this 
directory invoked? Hmmm... Maybe they're invoked from that 
`/usr/lib/cron/run-crons' script. Yes!! That's where it's invoked from. Yet 
another big, incomprehensible script. The core of that script is the following 
Bourne-shell loop. 

SPOOL=/var/spool/cron/lastrun
for CRONDIR in /etc/cron.{hourly,daily,weekly,monthly} ; do
    test -d $CRONDIR || continue
    BASE=${CRONDIR##*/}
    test -e $SPOOL/$BASE && {
        case $BASE in
          cron.hourly)  TIME="-cmin  +60 -or -cmin  60" ;;
          cron.daily)   TIME="-ctime +1  -or -ctime 1"  ;;
          cron.weekly)  TIME="-ctime +7  -or -ctime 7"  ;;
          cron.monthly) TIME="-ctime +`date -u +%d`"    ;;
        esac
        eval find $SPOOL/$BASE $TIME | xargs -r rm -f
    }
    if test ! -e $SPOOL/$BASE ; then
        touch $SPOOL/$BASE

        # keep going when something fails
        set +e
        for SCRIPT in $CRONDIR/*[^~,] ; do
            test -d $SCRIPT && continue
            test -x $SCRIPT || continue
           case "$SCRIPT" in
               *.rpm*)         continue ;;
               *.swap)         continue ;;
               *.bak)          continue ;;
               *.orig)         continue ;;
               \#*)            continue ;;
           esac
           /sbin/checkproc $SCRIPT && continue
           nice -15 $SCRIPT
        done
    fi
done

Now what does this mean? 
The command `BASE=${CRONDIR##*/}' means that BASE is set to the `longest 
substring of $CRONDIR which matches pattern "*/"', according to my Bash 
reference card. This just means that the leading path components are removed. 
(This is done more simply in C-shell!) In the case of the daily cron job, if 
there is a file /var/spool/cron/lastrun/cron.daily (which is true), then the 
following command is run. 
eval find /var/spool/cron/lastrun/cron.daily -ctime +1 -or -ctime 1 | xargs -r
rm -f

The `xargs' command (which I have never seen before) builds and executes a 
command line from standard input. Wierd! The `xargs' manual says this. 

If  the  standard  input  does not contain any non-blanks,
do not run the command.  Normally, the command is run once
even if there is no input.

So in this case, the command `rm -f' is executed for each of the files with 
modification times within 24 hours of the current time. I don't really 
understand this. 

It looks like the file removal commands in the file /etc/crontab are designed 
to synchronise the operation of the quarter-hour operations. The commands are 
only executed if the files in /var/spool/cron/lastrun have been removed. What 
a convoluted way of achieving a simple objective!! 

Next any file in the directory /etc/cron.daily which do not end in the 
characters `~' or `,' (presumed to be edited files) are executed if they are 
executable and do not have the endings .rpm, .swap, .bak or .orig as follows. 

           /sbin/checkproc $SCRIPT && continue
           nice -15 $SCRIPT

The loop continues is the process is already running. Otherwise it is run with 
nice level 15. 

What this all means finally is that any script in the directory
/etc/cron.daily 
will be run at 00:00 on each day. More to the point, the /etc/cron.monthly 
scripts are run at 00:00 at the beginning of each month. It all looks a bit 
dodgy because the `date' test is `date -u', which gives the current UTC day. 
But this should work, although the motivation for the `-u' is not quite clear. 

All I have to do now is write a script and put it in /etc/cron.monthly. No
problem! 

--------------------------------------------------------------------------------
sample methods:

# system-specific logs may be configured here 
 ## rotate apache logs weekly, keeping 8 rotations
 /www/apache/var/logs/*log {
     rotate 8
     postrotate
       kill -HUP `cat /www/apache/var/logs/httpd.pid`
     endscript
 }
 ## rotate rembo logs weekly, keeping 4 rotations
 /opt/rembo/logs/*.log {
     rotate 4
     postrotate
       /etc/rc.d/init.d/rembo stop
       /etc/rc.d/init.d/rembo start
     endscript
 }

--------------------------------------------------------------------------------
the roach method (for virtual hosting):

I have many virtual hosts.  I feared that apache would be start/stop
frequently,
once per virtual host, during the rotation process.  Therefore, I decided to
use
the 'copytruncate' option.  With this option the apache web server does not 
require a restart. -roach

/usr/apache/log/*log {
    weekly
    rotate 52
    notifempty
    missingok
    copytruncate
}

/usr/apache/log/dynahost.dbw.org/*log {
    weekly
    rotate 52
    notifempty
    missingok
    copytruncate
}

--------------------------------------------------------------------------------
Configure the new /etc/logrotate.d/apache file
Now Apache logs files residing in the /chroot/var/log/httpd directory instead
of 
/var/log/httpd and for this reason we need to modify the
/etc/logrotate.d/httpd 
file to point to the new chrooted directory. Also, we've compiled Apache with 
mod_ssl, so we'll add one more line to permit the logrotate program to rotate 
the ssl_request_log and ssl_engine_log files. Configure your 
/etc/logrotate.d/apache file to rotate your log files each week automatically. 

Create the apache file, touch /etc/logrotate.d/apache and add:         
/chroot/httpd/var/log/httpd/access_log {
        missingok
        postrotate
        /usr/bin/killall -HUP /chroot/httpd/usr/sbin/httpd
        endscript
        }

        /chroot/httpd/var/log/httpd/error_log {
        missingok
        postrotate
        /usr/bin/killall -HUP /chroot/httpd/usr/sbin/httpd
        endscript
        }

        /chroot/httpd/var/log/httpd/ssl_request_log {
        missingok
        postrotate
        /usr/bin/killall -HUP /chroot/httpd/usr/sbin/httpd
        endscript
        }

        /chroot/httpd/var/log/httpd/ssl_engine_log {
        missingok
        postrotate
        /usr/bin/killall -HUP /chroot/httpd/usr/sbin/httpd
        endscript
        }

--------------------------------------------------------------------------------
Credits and Quasi-bibliography:

The majority of this text is from:
http://www.topology.org/linux/logrotate.html
Reference: Alan Kennington's 

Some information was gathered from:
http://misc.epfl.ch/rembo/logrotate_conf.html
Reference: Unknown

Securing and Optimizing Linux: RedHat Edition -A Hands on Guide 
http://www.tldp.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/chap29sec258.html

Anything else was created by:
roach

date: Sun May 12 18:03:33 CDT 2002
--------------------------------------------------------------------------------

Logrotate: error reading top line of /var/lib/logrotate.status

If you get an error saying

/etc/cron.daily/logrotate:

error: error reading top line of /var/lib/logrotate.status

delete /var/lib/logrotate.status, and then run logrotate once with -f flag,
like

 #> logrotate -f /etc/logrorate.d/syslog

This should initialize the status file and the error should not be repeating.

Virtual Host Log Rotation for Apache

Many linux distributions include a utility called logrotate.d which is installed and configured by default. It is configured to rotate the main Apache log file typically weekly or monthly. Yet, Apache configuration for virtual hosts (virtual_host) suggests creating separate logs for each virtual host, and in a directory named for the site. This is typically under the primary Apache log directory. Consider a server where /var/log/apache2/access.log is the primary apache log, but there is a virtual host which a log under /var/log/apache2/www.smokinghotbabes.com/access.log. Now looking in /etc/logrotate.d/httpd the default config is:

/var/log/apache2/*.log {
    missingok
    notifempty
    postrotate 
	/usr/bin/killall -HUP httpd
    endscript
}

This default installed configuration for logrotate.d/httpd will only rotate the primary web site. It will not rotate virtual host sites. This can be amended by adding another part to the log rotation configuration script:

/var/log/apache2/*.log {
    missingok
    notifempty
    postrotate 
	/usr/bin/killall -HUP httpd
    endscript
}

/var/log/apache2/*/*.log {
    missingok
    notifempty
    postrotate 
	/usr/bin/killall -HUP httpd
    endscript
}

configuration testing and some important flags

Verbose

-v tells logrotate to say what it’s doing while it’s doing it. It’s very useful when trying to find out why logrotate doesn’t rotate a log when you want it to.

Debug

-d tells logrotate to go through the motions of rotating logs but not actually rotate them. Good for checking that the config file is formatted properly and that logrotate can find the log files it would rotate. It doesn’t actually run the rotations and it doesn’t test some parts of the process like the postrotate scripts.

Force

-f forces logrotate to rotate all logs, whether or not they would normally need to be rotated at that time.

External Resources

• HowTo: The Ultimate Logrotate Command Tutorial with 10 Examples
• Apache logs and logrotate.d
• Understanding logrotate - part 2