|
Home | Switchboard | Unix Administration | Red Hat | TCP/IP Networks | Neoliberalism | Toxic Managers |
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix |
|
Log rotation mechanism in RHEL/Centos/Oracle linux is based on utility logrotate which is installed by default.
|
Default configuration file for logrotate for Red Hat 5.x is as following
# see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # RPM packages drop log rotation information into this directory include /etc/logrotate.d # no packages own wtmp -- we'll rotate them here /var/log/wtmp { monthly minsize 1M create 0664 root utmp rotate 1 } /var/log/btmp { missingok monthly minsize 1M create 0600 root utmp rotate 1 } # system-specific logs may be also be configured here.
As you can see by default logs are rotated weekly and kept four weeks with no compression. If additional packages are installed from RPMs, it can drop include for log rotation in /etc/logrotate.d . But few packages do. So it's up to sysadmin to implement relocation properly. As you can see default settings are pretty unsatisfactory too.
The first think to understand this packages install logrotate includes in /etc/logrotate.d and which don't. After a typical installation of RHEL the directory logrotate.d contains a dozen of scripts for already installed packages. For example:
-rw-r--r-- 1 root root 144 Dec 17 2009 acpid -rw-r--r-- 1 root root 288 Jun 28 2007 conman -rw-r--r-- 1 root root 71 Jan 19 2011 cups -rw-r--r-- 1 root root 167 Oct 6 12:12 httpd -rw-r--r-- 1 root root 571 Aug 21 2006 mgetty -rw-r--r-- 1 root root 136 Aug 12 2008 ppp -rw-r--r-- 1 root root 442 Jul 16 2008 psacct -rw-r--r-- 1 root root 61 Sep 30 05:41 rpm -rw-r--r-- 1 root root 232 Jul 29 2011 samba -rw-r--r-- 1 root root 121 May 21 2009 setroubleshoot -rw-r--r-- 1 root root 543 Feb 16 2010 squid -rw-r--r-- 1 root root 71 Jun 6 2011 subscription-manager -rw-r--r-- 1 root root 306 Jan 5 2010 syslog -rw-r--r-- 1 root root 48 Aug 23 2006 tux -rw-r--r-- 1 root root 32 Oct 26 2009 up2date -rw-r--r-- 1 root root 188 Mar 28 2011 vsftpd.log -rw-r--r-- 1 root root 100 Dec 14 2009 wpa_supplicant -rw-r--r-- 1 root root 100 Jun 14 2011 yum
The most important of them is syslog. It is pretty small, less the 10 lines:
syslog /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron { sharedscripts postrotate /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true endscript }
If you modify rsyslog setup for example by creating a separate log file for critical messages only you need to add it into this script.
Option sharedscripts blocks repeated invocation of the daemon for each log specified. Here is how this item semantic is defined in the man page:
Normally, prerotate and postrotate scripts are run for each log which is rotated and the absolute path to the log file is passed as first argument to the script. That means a single script may be run multiple times for log file entries which match multiple files (such as the /var/log/news/* example). If sharedscripts is specified, the scripts are only run once, no matter how many logs match the wildcarded pattern, and whole pattern is passed to them. However, if none of the logs in the pattern require rotating, the scripts will not be run at all. If the scripts exit with error, the remaining actions will not be executed for any logs. This option overrides the nosharedscripts option and implies create option.
Apache logs rotation is performed using the following settings:
httpd /var/log/httpd/*log { missingok notifempty sharedscripts postrotate /sbin/service httpd reload > /dev/null 2>/dev/null || true endscript }
Default cron script for invocation of logrotate is /etc/cron.daily/logrotate:
/usr/sbin/logrotate /etc/logrotate.conf EXITVALUE=$? if [ $EXITVALUE != 0 ]; then /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]" fi exit 0
That means that log rotation is scheduled for midnight of each day. Usually this is OK, but for central syslog server you might need more frequent invocation.
In default configuration a pre-determined number of important log files (such as /var/log/messages ) are kept on disk for a very limited time, and older logs are deleted automatically. By default, logs are kept for just 4 weeks, which is unacceptable for production servers and needs to be changed during the initial configuration of the server.
By default, logs are kept for just 4 weeks, which is unacceptable for production servers and needs to be changed during the initial configuration of the server. |
To change the time your logs are kept for, edit /etc/logrotate.conf and change the rotate parameter. You can also override this for specific logs by editing the files in /etc/log/logrotate.d/ directory. For more details see the logrotate man page.
Old logfiles are typically compressed with gzip or bzip (the latter provides better compression of text files) to save space. They also can be moved to a new directory (see options compress, delaycompress below).
Let's assume we have a new service "foobar" which is configured to write logs to the directory /var/logs/foobar . To accommodate rotation needs for this daemon we need to add logrotate statement in /etc/logrotate.conf describing rotation parameters. For example:
/var/log/foobar/*.log {
monthly
missingok
rotate 6
compress
delaycompress
create 640 root root
mail root@localhost
sharedscripts
postrotate
/etc/init.d/foobar restart
endscript
}
Of course it can be done simpler, but let's discuss this more complex example.
Here we use monthly to ensure that the logs are rotated each month and rotate 6 to make sure no more than seven previous logs are kept. Now an exotic functionality: when a log expires, it's emailed to root@localhost before being deleted as specified by mail. that can be done for daily logs. Otherwise email might be too big.
compress shrinks rotated logs except previous generation because of delaycompress. Using compress with delaycompress is a recommended setting.
Logrotate doesn't complain if the log files are missing thanks to missingok. After the logs have been rotated, a new one is created by create as being owned by the user and group root and the service is restarted.
compress | This is used to compress the rotated log file with gzip. |
nocompress | This is used when you do not want to compress rotated log files. |
copytruncate | This is used when processes are still writing information to open log files. This option copies the active log file to a backup and truncates the active log file. |
nocopytruncate | This copies the log files to backup, but the open log file is not truncated. |
create mode owner group | This rotates the log file and creates a new log file with the specified permissions, owner, and group. The default is to use the same mode, owner, and group as the original file. |
nocreate | This prevents the creation of a new log file. |
delaycompress | When used with the compress option, the rotated log file is not compressed until the next time it is cycled. |
nodelaycompress | This overrides delaycompress. The log file is compressed when it is cycled. |
errors address | This mails logrotate errors to an address. |
ifempty | With this, the log file is rotated even if it is empty. This is the default for logrotate. |
notifempty | This does not rotate the log file if it is empty. |
mail address | This mails log files that are cycled to an address. When mail log files are cycled, they are effectively removed from the system. |
nomail | When mail log files are cycled, a copy is not mailed. |
olddir directory | With this, cycled log files are kept in the specified directory. This directory must be on the same filesystem as the current log files. |
noolddir | Cycled log files are kept in the same directory as the current log files. |
prerotate/endscript | These are statements that enclose commands to be executed prior to a log file being rotated. The prerotate and endscript keywords must appear on a line by themselves. |
postrotate/endscript | These are statements that enclose commands to be executed after a log file has been rotated. The postrotate and endscript keywords must appear on a line by themselves. |
daily | This is used to rotate log files daily. |
weekly | This is used to rotate log files weekly. |
monthly | This is used to rotate log files monthly. |
rotate count | This specifies the number of times to rotate a file before it is deleted. A count of 0 (zero) means no copies are retained. A count of 5 means five copies are retained. |
tabootext [+] list | This directs logrotate to not rotate files with the specified extension. The default list of extensions is .rpm-orig, .rpmsave, v, and ~. |
size size | With this, the log file is rotated when the specified size is reached. Size may be specified in bytes (default), kilobytes (sizek), or megabytes (sizem). |
You can also use this utility for rotation of simple backups and baselines. For example you can rotates
files with the .tgz
extension which are located in the /var/data/backups
directory.
The parameters in this file will override the global defaults in the /etc/logrotate.conf
file. In this case, the rotated files won't be compressed, they'll be held for 30 days only if they
are not empty, and they will be given file permissions of 600 for user root
.
/var/data/etcbackups/*.tgz { daily rotate 30 nocompress missingok notifempty create 0600 root root }This was you can simply put tar command in cron and logrotate will ensure preservation of 30 last copies of /etc/ directory. This way you can avoid writing script that take care of all those actions.
While writing and testing slices for new services very useful is the debug option:
logrotate -d file
This will show you what logrotate will do, without actually rotating anything, and this capability in not only valuable while testing you own /etc/logrotate.conf file and slices that you create, but also understanding logrotate actions on a new server that you need to manage.
Also logrotate -f file will force the rotation even if it normally should not happen (usually logrotate runs and rotate logs once per day from cron). This is also useful option if you experience crash or something that requires "a fresh start" of log files for troubleshooting.
If you want logrotate to execute only a specific configuration file, and not all of them, then issue the logrotate command with just that filename as the argument like this:
logrotate -f /etc/logrotate.d/syslog
In other words -f switch forces the logs to be rotated.
-f switch forces the logs to be rotated.
-d switch is the debug option which will show you what logrotate will do, without actually rotating anything |
It might be useful to run logrotate with this switch every time the configuration files are changed.
As we mentioned before there is great sense to use bzip2 for compression of large log file. This can be achieved by using statements
compresscmd /bin/bzip2 compressext .bz2>
For example:
/var/log/cron { size 10M copytruncate create compress compresscmd /bin/bzip2 compressext .bz2 rotate 4 }
|
Switchboard | ||||
Latest | |||||
Past week | |||||
Past month |
Many of the services installed on Linux machines will produce logfiles which grow, and grow, and grow. If left unchecked you can easily fill a disk with a large collection of logfiles if you're not careful.The most common method of keeping logfile growth in check is to use logrotate, and many Debian packages are setup to work with this by default.
The most obvious package which uses it is Apache, the webserver, which by default keeps its logfiles in the directory /var/log/apache (or /var/log/apache2).
If you examine this directory you will see that there are a bunch of logfiles which are archived:
root@mystery:~# ls -1 /var/log/apache2/ access.log access.log.1 access.log.2.gz access.log.3.gz access.log.4.gz access.log.5.gz error.log error.log.1 error.log.2.gz error.log.3.gz error.log.4.gz error.log.5.gzHere the current logfiles access.log, error.log are kept raw as are yesterday's logfiles (access.log.1 and error.log.1). Previous logfiles are compressed with gzip and only kept for five weeks. (I know it's five weeks and not five days because I've looked at the configuration - It's not clear from this output though!)
The process that is in charge of compressing and rotating these logfiles is called logrotate and it is executed once per day upon Debian installations.
As we saw when we were looking at scheduling commands with cron there is a directory called /etc/cron.daily which contains scripts which are executed once per day. Here you will find the logrotate driver script.
Every day this script runs and examines two things:
- The configuration file /etc/logrotate.conf
- The configuration directory /etc/logrotate.d
The latter is where most of our packages are configured. This directory contains configuration files which other packages have installed. For example if you install apache the file /etc/logrotate.d/apache will be installed.
Many servers such as exim the mailserver will install their own configuration file, and you can add your own.
A typical logrotate configuration file looks like this:
/var/log/apache/*.log { weekly missingok rotate 52 compress delaycompress notifempty create 640 root adm sharedscripts postrotate if [ -f /var/run/apache.pid ]; then /etc/init.d/apache restart > /dev/null fi endscript }You can see several important things here. The most obvious is the list of files that will be matched by this configuration file:
/var/log/apache/*.log { ... }After this we have a collection of configuration terms, a different one on each line. In the example above we have:
- weekly
- The files should be rotated every week. Opposite: daily
- rotate nn
- We should keep no more than nn files.
- compress
- Compress older files with gzip. Opposite: nocompress
- delaycompress
- Don't compress yesterdays files. Opposite: compress
- notifempty
- Don't do any rotation if the logfile is empty. Opposite: ifempty
- create xx user group
- If we have to create the new file give it the given mode, owner, and group.
- sharedscripts
- Run any given prerotate or postrotate script for each logfile individually. Opposite: nosharedscripts.
- postrotate + endscript
- Anything between these is executed after the rotation process. Opposite : prerotate
Hopefully that should have made sense!
The upshot of this script is that any file which matches /var/log/apache/*.log is rotated every week, compressed, if it's non-empty. The new file is created with the file mode of 640, and after the rotation has finished the server is restarted.
If we wish to install a local service which creates a logfile we can cause it to be rotated very easily, just by adding a new logrotate configuration file.
Assuming we have a new service "fred" which produces its output in /var/log/fred/output.log we can cause this to be rotated every day with a script like this:
/var/log/fred/*.log { daily missingok rotate 7 compress delaycompress create 640 fred fred sharedscripts /etc/init.d/fred restart endscript }This will:
- Run daily.
- Keep no more than 7 days worth of logfiles at any one time.
- Not complain if there is a logfile missing.
- Compress the older files, but not yesterdays.
- Create the new logfiles as being owned by the user and group fred.
- Restart the service after rotating the logfiles.
Any of the existing files in the logrotate directory can be examined for more examples - and the manpage documents all the options you may use in a clear manner:
man logrotate
March 22, 2008
On a default install of CentOS or Red Hat Enterprise Linux, the log rotation script will automatically rotate the Apache log file each day and then reload the httpd service. This post looks at how to prevent this action from occuring automatically, or to change the behaviour to rotate the log files if your naming convention for log files is different from the default.
The cron daemon on a CentOS or Red Hat Enterprise Linux server by default runs the scripts in the directory /etc/cron.daily on a daily basis. This includes running the "logrotate" script which runs /usr/sbin/logrotate passing it the /etc/logrotate.conf configuration file. The default version of this logrotate.conf file is as follows:
# see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4
# create new (empty) log files after rotating old ones create
# uncomment this if you want your log files compressed #compress
# RPM packages drop log rotation information into this directory include /etc/logrotate.d
# no packages own wtmp -- we'll rotate them here /var/log/wtmp { monthly minsize 1M create 0664 root utmp rotate 1 }
# system-specific logs may be also be configured here.These default variables are fairly straight forward and give you control over how frequently the log files are rotated, how many lots of previous log files to keep before deleting them, whether or not to compress them and so on. It also tells logrotate to process all the files in the directory /etc/logrotate.d
The file which controls log rotation of Apache on CentOS and Red Hat Enterprise Linux is named /etc/logrotate.d/httpd, and the default contents of this file are as follows:
/var/log/httpd/*log { missingok notifempty sharedscripts postrotate /sbin/service httpd reload > /dev/null 2>/dev/null || true endscript }What this file is saying, is to rotate all files that match the pattern /var/log/httpd/*.log. The default access and error log files in Apache on CentOS and RHEL are named "access_log" and "error_log" so they match this pattern.
If you have changed the default names of the log files and they don't match this pattern, then you can have logrotate rotate your log files by making sure the pattern is in this file in place of the /var/log/httpd/*.log line. For example, if your log files were named error.foo and access.foo then /var/log/httpd/*.foo would match them.
If you do not wish the logrotate script to rotate your Apache log files at all, then you can simply delete this file like so, logged in either as root or using sudo:
rm -f /etc/logrotate.d/httpdThe changes will take effect the next time the logrotate script is run.
December 27, 2000 | TechRepublic
Log files are the most valuable tools available for Linux system security. The logrotate program is used to provide the administrator with an up-to-date record of events taking place on the system. The logrotate utility may also be used to back up log files, so copies may be used to establish patterns for system use. In this Daily Drill Down, I'll cover the following topics:
;
- The logrotate configuration
- Setting defaults for logrotate
- Using the include option to read other configuration files
- Setting rotation parameters for specific files
- Using the include option to override defaults
The logrotate program
The logrotate program is a log file manager. It is used to regularly cycle (or rotate) log files by removing the oldest ones from your system and creating new log files. It may be used to rotate based on the age of the file or the file's size, and usually runs automatically through the cron utility. The logrotate program may also be used to compress log files and to configure e-mail to users when they are rotated.The logrotate configuration
The logrotate program is configured by entering options in the /etc/logrotate.conf file. This is a text file, which may contain any of the configuration options listed in the table below. The options entered in /etc/logrotate.conf may be used to set configuration parameters for any log file on the system. These options may also be used to allow logrotate to read configuration parameters from other log files, by using the include parameter.;
; Options
; Function compress This is used to compress the rotated log file with gzip. nocompress This is used when you do not want to compress rotated log files. copytruncate This is used when processes are still writing information to open log files. This option copies the active log file to a backup and truncates the active log file. nocopytruncate This copies the log files to backup, but the open log file is not truncated. create mode owner group This rotates the log file and creates a new log file with the specified permissions, owner, and group. The default is to use the same mode, owner, and group as the original file. nocreate This prevents the creation of a new log file. delaycompress When used with the compress option, the rotated log file is not compressed until the next time it is cycled. nodelaycompress This overrides delaycompress. The log file is compressed when it is cycled. errors address This mails logrotate errors to an address. ifempty With this, the log file is rotated even if it is empty. This is the default for logrotate. notifempty This does not rotate the log file if it is empty. mail address This mails log files that are cycled to an address. When mail log files are cycled, they are effectively removed from the system. nomail When mail log files are cycled, a copy is not mailed. olddir directory With this, cycled log files are kept in the specified directory. This directory must be on the same filesystem as the current log files. noolddir Cycled log files are kept in the same directory as the current log files. prerotate/endscript These are statements that enclose commands to be executed prior to a log file being rotated. The prerotate and endscript keywords must appear on a line by themselves. postrotate/endscript These are statements that enclose commands to be executed after a log file has been rotated. The postrotate and endscript keywords must appear on a line by themselves. daily This is used to rotate log files daily. weekly This is used to rotate log files weekly. monthly This is used to rotate log files monthly. rotate count This specifies the number of times to rotate a file before it is deleted. A count of 0 (zero) means no copies are retained. A count of 5 means five copies are retained. tabootext [+] list This directs logrotate to not rotate files with the specified extension. The default list of extensions is .rpm-orig, .rpmsave, v, and ~. size size With this, the log file is rotated when the specified size is reached. Size may be specified in bytes (default), kilobytes (sizek), or megabytes (sizem).
The /etc/logrotate.conf file
The /etc/logrotate.conf file is the default configuration file for logrotate. The default /etc/logrotate.conf file installed with Red Hat Linux is shown below:
# see "man logrotate" for details
# rotate log files weekly
weekly# keep 4 weeks worth of backlogs
rotate 4# send errors to root
errors root
# create new (empty) log files after rotating old ones
create# uncomment this if you want your log files compressed
#compress
1
# RPM packages drop log rotation information into this directory
include /etc/logrotate.d# no packages own lastlog or wtmp --we'll rotate them here
/var/log/wtmp {
; monthly
; create 0664 root utmp
; rotate 1
}/var/log/lastlog {
; monthly
; rotate 1
}# system-specific logs may be configured here
Setting defaults for logrotate
Default configuration settings are normally placed close to the beginning of the logrotate.conf file. These settings are usually in effect system-wide. The default settings for logrotate on this system are established in the first 12 lines of the file.The third line
weeklyspecifies that all log files will be rotated weekly.
The fifth line
rotate 4specifies that four copies of old log files are retained before the files are cycled. Cycling refers to removing the oldest log files and replacing them with new copies.
The seventh line
errors rootsends all logrotate error messages to root.
The ninth line
createconfigures logrotate to automatically create new log files. The new log files will have the same permissions, owner, and group as the file being rotated.
The eleventh line
#compressprevents logrotate from compressing log files when they are rotated. Compression is enabled by removing the comment (#) from this line.
Using the include option
The include option allows the administrator to take log file rotation information, which may be installed in several files, and use it in the main configuration file. When logrotate finds the include option on a line in logrotate.conf, the information in the file specified is read as if it appeared in /etc/logrotate.conf.Line 13 in /etc/logrotate.conf
include /etc/logrotate.dtells logrotate to be read in the log rotation parameters, which are stored in the files contained in the /etc/logrotate.d directory. The include option is very useful when RPM packages are installed on a system. RPM packages' log rotation parameters will typically install in the /etc/logrotate.d directory.
The include option is important. Some of the applications that install their log rotation parameters to /etc/logrotate.d by default are apache, linuxconf, samba, cron, and syslog. The include option allows the parameters from each of these files to be read into logrotate.conf.
Using the include option in /etc/logrotate.conf allows the administrator to configure a rotation policy for these packages through a single configuration file.
Using include to override defaults
When a file is read by /etc/logrotate.conf, the rotation parameters specified in the include will override the parameters specified in the logrotate file. An example of /etc/logrotate.conf being overridden is shown below:
#Log rotation parameters for linuxconf
/var/log/htmlaccess.log
{ errors jim
notifempty
nocompress
weekly
prerotate
/usr/bin/chattr -a /var/log/htmlaccess.log
endscript
postrotate
/usr/bin/chattr +a /var/log/htmlaccess.log
endscript
}
/var/log/netconf.log
{ nocompress
monthly
}In this example, when the /etc/logrotate.d/linuxconf file is read by /etc/logrotate.conf, the following options will override the defaults specified in /etc/logrotate.conf:
Notifempty
errors jimThe nocompress and weekly options do not override any options contained in /etc/logrotate.conf.
Setting parameters for a specific file
Configuration parameters for a specific file are often required. A common example would be to include a section in the /etc/logrotate.conf file to rotate the /var/log/wtmp file once per month and keep only one copy of the log. When configuration is required for a specific file, the following format is used:
#comments
/full/path/to/file
{
option(s)
}The following entry would cause the /var/log/wtmp file to be rotated once a month, with one backup copy retained:
#Use logrotate to rotate wtmp
/var/log/wtmp
{
monthly
rotate 1
}
Although the opening bracket may appear on a line with other text or commands, the closing bracket must be on a line by itself.
Using the prerotate and postrotate options
The section of code below shows a typical script in /etc/logrotate.d/syslog. This section applies only to /var/log/messages. On a production server, /etc/logrotate.d/syslog would probably contain similar entries.
/var/log/messages
{
prerotate
/usr/bin/chattr -a /var/log/messages
endscript
postrotate
/usr/bin/kill -HUP syslogd
/usr/bin/chattr +a /var/log/messages
endscript
}The format for this script uses the following methods:
;
- The first line, /var/logmessages, declares the file for which this script will be used.
- The curly braces,{ }, are used to enclose the entire script. All commands contained within these braces will be run on the /var/log/messages file.
- The prerotate command specifies actions to be taken prior to the file being rotated by logrotate.
- The command /usr/bin/chattr -a is run to remove the append-only attribute from /var/log/messages.
- The endscript command marks the end of the prerotate portion of this script.
- The next line, postrotate, specifies the following commands are to be run on /var/log/messages after the file has been rotated by logrotate.
- The command /usr/bin/killall -HUPsyslogd is run to reinitiate the system logging daemon, syslogd.
- The next command, /usr/bin/chattr +a /var/log/messages, reassigns the append-only attribute to the /var/log/messages file. This means the file may only be seen in append mode. This prevents the file from being overridden by any other program or user.
- The endscript command appears on a line by itself and marks the end of the postrotate portion of this script.
- The last curly brace,}, marks the end of commands to be applied to the /var/log/messages file.
Running logrotate
There are three steps involved in running logrotate:
;
- Identify the log files on your system.
- Create rotation schedules and parameters for the log files.
- Run logrotate through the cron daemon.
The code below shows the default cronjob shipped with Red Hat Linux to allow logrotate to run daily:
#/etc/cron.daily/logrotate
#! /bin/sh/usr/sbin/logrotate /etc/logrotate.conf
This cronjob allows logrotate to run daily with the rotation parameter specified in /etc/logrotate.conf.
Conclusion
Log rotation is the first step in log file management. The logrotate utility provides the Linux administrator with the ability to maintain a log file rotation policy and to retain copies of log files to assist in establishing patterns related to system usage. In this Daily Drill Down, we looked at the installation and configuration of logrotate, used the include option to read configuration files related to RPM packages, and ran logrotate as a cronjob. We also discussed the proper methods for restarting logrotate after the log rotation procedure is completed.
The authors and editors have taken care in preparation of the content contained herein but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.
Understanding logrotate on RHEL - part 1
It's no fun when log files grow out of control. In this two-part series, learn how to use logrotate to keep those logs in check.
What is logrotate?
It may surprise you to learn that logrotate is a program used to rotate logs. It's true! The system usually runs logrotate once a day, and when it runs it checks rules that can be customized on a per-directory or per-log basis.
"Log rotation" refers to the practice of archiving an application's current log, starting a fresh log, and deleting older logs. And while we're explaining things, a "log" is a file where an application stores information that might be useful to an administrator or developer - what it's been doing, what errors it's run into, that sort of thing. So logs are good, you just usually don't want to keep a ton of them around. That's where logrotate comes in.
The importance of log rotation
Logs are wonderful things when you want to track usage or troubleshoot an application. Unfortunately the more information that gets logged, the more disk space the log uses. Over time it can really add up.
A log left unrotated can grow to a pretty unwieldy size. Running out of disk space because of a giant log is a problem of course, but a huge log file can also slow down the process of resizing or backing up your virtual server. Another practical consideration is that it's hard to look for a particular event if you have a million log entries to skim through. So on the whole it's a good idea to keep log files down to a manageable size, and to prune them when they get too old to be of much use.
Fortunately logrotate makes log rotation easy.
How it works
The system runs logrotate on a schedule, usually daily. In fact, you'll find the script that runs logrotate daily at:
/etc/cron.daily/logrotate
If you want logrotate to run more often (for hourly log rotation, for example) you'll need to look into using cron to run logrotate through a script in /etc/cron.hourly.
When logrotate runs it reads its configuration files to determine where to find the log files it needs to rotate, and to check on details like how often the files should be rotated and how many archived logs to keep.
logrotate.conf
The main logrotate configuration file is located at:
/etc/logrotate.conf
If you look inside that file you'll see the default parameters logrotate uses when it rotates logs. The file is nicely commented, so skim it to see how things are set up. We'll talk about several of the specific commands in that file shortly.
Note that one line reads:
include /etc/logrotate.d
That's where we'll find most of the application-specific configuration files.
logrotate.d
Take a look inside the directory where you'll store application-specific log settings:
ls /etc/logrotate.d
Depending on how much you've installed on your server there may be no files in this directory, or there may be several. In general, applications that are installed through Red Hat's package manager (yum) will also create a config file in /etc/logrotate.d.
Most likely you will at least see a config file for syslog, which logrotate will read when it goes to rotate the system logs. If you look inside you'll see an entry for various system logs along with some commands similar to what you saw in logrotate.conf.
Inside an application file
As an example, let's take a look at the contents of a logrotate config file that might be put in place when you install apache:
/var/log/httpd/*log { missingok notifempty sharedscripts postrotate /sbin/service httpd reload > /dev/null 2>/dev/null || true endscript }
We'll look at what most of the specific directives in this file mean in a bit, but the short version is that when logrotate runs it will check for any files in /var/log/httpd that end in "log" and rotate them, so long as they aren't empty. If it checks the httpd directory and doesn't find any logfiles it won't throw an error. Then it will run the command in the "postrotate/endscript" block (in this case, a command that will tell apache to restart), but only after it's processed all the specified logs.
What you don't see in that file are some settings you saw back in logrotate.conf. This is because the commands in logrotate.conf act as defaults for log rotation. You can specify different settings for any application where you want to override the defaults. For example, if you run a busy web server, you may want to include a "daily" command in apache's config block so apache's logs will rotate daily instead of the default weekly rotation.
That might be more clear if we talk about what some of the more commonly-used commands actually do in a logrotate config file. So let's do that next.
Configuration commands
You can get a full list of commands used in logrotate configuration files by checking the man page:
man logrotate
We'll go over more commonly-used commands here.
Remember, the config files for applications in /etc/logrotate.d inherit their defaults from the main /etc/logrotate.conf file.
Log files
A log file and its rotation behavior is defined by listing the log file (or files) followed by curly brackets. Most application configuration files will contain just one of these blocks, but it's possible to put more than one in a file, or to add log file blocks to the main logrotate.conf file.
You can list more than one log file for a block either by using a wildcard in the name or by separating log files in the list with spaces. For example, to specify all files in the directory /var/foo that end in ".log", as well as the file "/var/bar/log.txt", you would set up the block like so:
/var/foo/*.log /var/bar/log.txt { blah blah blah blah blah blah redux }
Just not with as many blahs.
Rotate count
The "rotate" command determines how many archived logs will be kept around before logrotate starts deleting the older ones. For example:
rotate 4
That command tells logrotate to keep 4 archived logs at a time. If there are already four archived logs when the log is rotated again, the oldest one (the one with ".4" at the end, usually) will be deleted to make room for the new archive.
Rotation interval
You can specify a command that will tell logrotate how often to rotate a particular log. The possible commands include:
daily weekly monthly yearly
If a rotation interval is not specified the log will be rotated whenever logrotate runs (unless another condition like "size" has been set).
If you want to use a time interval other than the keywords listed here you'll have to get clever with cron and a separate config file. For example, if you wanted to rotate a particular log file hourly, you could create a file in "/etc/cron.hourly" (you may need to create that directory too) that would contain a line like:
/usr/sbin/logrotate /etc/logrotate.hourly.conf
Then put the configuration for that hourly run of logrotate (the log file location, whether or not to compress old files, and so on) into "/etc/logrotate.hourly.conf".
Size
You can specify a file size that logrotate will check when determining whether or not to perform a rotation by using the "size" command. The format of the command tells logrotate what units you're using to specify the size:
size 100k size 100M size 100G
The first example would rotate the log if it gets larger than 100 kilobytes, the second if it's larger than 100 megabytes, and the third if it's over 100 gigabytes. I don't recommend using a limit of 100G, mind you, the example just got a little out of hand there.
The size command takes priority over a rotation interval. When a log rotates because it hit its size limit the time interval resets for that log file. For example, if a log set to rotate "weekly" is rotated because it hit its max size after four days, logrotate will wait another week after that before rotating the log based on the time interval that has passed. As a result, if you use an interval other than "daily" and specify a maximum size for the log file, the log rotation won't be guaranteed to happen on the same day every week.
Compression
If you want archived logfiles to be compressed (in gzip format) you can include the following command, usually in /etc/logrotate.conf:
compress
This is normally a good idea, since log files are usually all text, and text compresses very well. You might, however, have some archived logs you don't want compressed, but still want compression to be on by default. In those cases you can include the following command in an application-specific config:
nocompress
One more command of note in regard to compression is:
delaycompress
This command can be useful if you want the archived logs to be compressed, but not right away. With "delaycompress" active an archived log won't be compressed until the next time the log is rotated. This can be important when you have a program that might still write to its old logfile for a time after a fresh one is rotated in. Note that "delaycompress" only works if you also have "compress" in your config.
An example of a good time to use delaycompress would be when logrotate is told to restart apache with the "graceful" or "reload" directive. Since old apache processes would not be killed until their connections are finished, they could potentially try to log more items to the old file for some time after the restart. Delaying the compression ensures that you won't lose those extra log entries when the logs are rotated.
Postrotate
The "postrotate" script is run by logrotate each time it rotates a log specified in a config block. You'll usually want to use this to restart an application after the log rotation so the app can switch to a new log.
postrotate /usr/sbin/apachectl restart > /dev/null endscript
That "> /dev/null" bit at the end tells logrotate to pipe the command's output to, well, nowhere. Otherwise the output of that command will be sent off to the console or the log or email or whatever, and in this case, you don't really care about the output if everything restarted okay.
The "postrotate" command tells logrotate that the script to run will start on the next line, and the "endscript" command says that the script is done.
Sharedscripts
Normally logrotate will run the "postrotate" script every time it rotates a log. This is true for multiple logs using the same config block. So for example, a web server config block that refers to both the access log and the error log will, if it rotates both, run the "postrotate" script twice (once for each file rotated). So if both files are rotated, the web server will be restarted twice.
To keep logrotate from running that script for every log, you can include the command:
sharedscripts
That tells logrotate to wait until it's checked all the logs for that config block before running the postrotate script. If one or both of the logs get rotated, the postrotate script still only gets run once. If none of the logs get rotated, the postrotate script won't run at all.
Summary
You've seen an overview of what logrotate does and what kind of configuration options are available to you. You should be all set to go poking around in the existing configs and adapt them to your needs. But let's not stop there! In the next article we'll look at putting an example config together (to rotate the logs for custom virtual hosts), and also cover some handy troubleshooting approaches.
- -- Jered
Comments Chris Riney - June 7, 2006I'm glad that he mentioned that this was based on a Debian systems, since the RedHat & SuSE systems that I work with do not have the syslog "Feature" that is described (must be part of syslogNG which I'm not familiar with). Both of these major commercial distributions manage the syslog files via logrotate.
Actually I'm just starting to work with SuSE Enterprise, and I'm noticing the differences in the way RedHat & SuSE implement scheduling Hourly, Daily, Weekly, & Monthly rotations.
A feature of of LogRotate 3.7.x that I found, (and was looking for) is the 'dateext' which appends the rotation date to the end instead of a sequential number. The problem I have with the sequential number, is that logrotate moves the older log's up the sequence, which causes those file to be backed-up multiple times with differential backups.
One item that you didn't cover, is how logrotate knows when it is the proper time to rotate a daily, weekly, or monthly file. Part of that is controlled by the status file (/var/lib/logrotate.status on most Linux system) which tracks when a particular file was last rotated. The other part is that Monthly rotations are done with the first run of logrotate that month, and Weekly's are run if the last run was later in the previous week, or the last rotation was over 7 days prior (Sunday=0 Saturday=6). So if you are running logrotate daily, then it works itself out to the first day of the month, and Sunday's for the weeklies.
These are my personal notes on how to use `logrotate' on a linux system.
I'm writing these notes to help me make sense of how things work.2005-5-5: I don't know where the logrotate source went to. All of my links to logrotate are broken. The source in about September 2001 was logrotate-3.3.tar.gz. That probably isn't very useful though.
How logrotate is invoked from cron.
On one of my RedHat 5.2 systems, `crond' is started from the script /etc/rc.d/init.d/crond. The manual `man cron' says that cron
searches for /etc/crontab and the files in the /etc/cron.d/ directory.My /etc/crontab file contains the following.
SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root # run-parts 01 * * * * root run-parts /etc/cron.hourly 02 4 * * * root run-parts /etc/cron.daily 22 4 * * 0 root run-parts /etc/cron.weekly 0 0 1 * * root run-parts /etc/cron.monthlyAnd the /etc/cron.d directory contains nothing.So clearly, any logrotate commands are being run out of one or more of the /etc/cron.* files. The manual `man 5 crontab' indicates that the above commands mean that the program `run-parts' will be run as user `root' for each of the scripts.
The command `run-parts' turns out to be the script /usr/bin/run-parts, which is very short, as follows.
#!/bin/bash # run-parts - concept taken from Debian # keep going when something fails set +e if [ $# -lt 1 ]; then echo "Usage: run-parts <dir>" exit 1 fi if [ ! -d $1 ]; then echo "Not a directory: $1" exit 1 fi for i in $1/* ; do [ -d $i ] && continue if [ -x $i ]; then $i fi done exit 0In essence, this just runs all of the scripts in the specified directory. E.g. each hour, the executable plain files in /etc/cron.hourly are run.It turns out that the `logrotate' program is invoked from /etc/cron.daily. This is the contents of the file /etc/cron.daily/logrotate.
#!/bin/sh /usr/sbin/logrotate /etc/logrotate.confThis means that the only file that `logrotate' gets its instructions from directly is /etc/logrotate.conf, which contains the following lines.# see "man logrotate" for details # rotate log files weekly weekly # keep 8 weeks worth of backlogs rotate 8 # send errors to root errors root # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # RPM packages drop log rotation information into this directory include /etc/logrotate.d # no packages own lastlog or wtmp -- we'll rotate them here /var/log/wtmp { monthly rotate 1 } # system-specific logs may be configured hereThis is all explained in the manual `man logrotate'.Operation of logrotate.
The directory /etc/logrotate.d contains the following files.
-rw-r--r-- 1 root root 354 Oct 13 1998 apache -rw-r--r-- 1 root root 108 Aug 28 1999 cron -rw-r--r-- 1 root root 188 Oct 14 1998 linuxconf -rw-r--r-- 1 root root 156 Oct 13 1998 mgetty -rw-r--r-- 1 root root 327 Aug 12 1998 syslog -rw-r--r-- 1 root root 457 Sep 10 1998 uucpThe file /etc/logrotate.d/apache is the one I'm interested in for this exercise. This file contains the following./var/log/httpd/access_log { postrotate /usr/bin/killall -HUP httpd endscript } /var/log/httpd/agent_log { postrotate /usr/bin/killall -HUP httpd endscript } /var/log/httpd/error_log { postrotate /usr/bin/killall -HUP httpd endscript } /var/log/httpd/referer_log { postrotate /usr/bin/killall -HUP httpd endscript }When I installed the latest version of Apache to get the PHP3 and PostgreSQL to work (around 20 March 2000) on my web server machine, I installed Apache so that the log files were in /home2/apache/logs instead of /var/log/httpd.Therefore what I need to do now is to modify the /etc/logrotate.d/apache file so that the files referred to are all in the directory /home2/apache/logs instead. My new /etc/logrotate.d/apache script is as follows, and I saved the old one in directory /etc/logrotate.d/old1.
# The new improved logrotate script for apache on fox. /home2/apache/logs/*-access_log { rotate 9 monthly errors [email protected] create ifempty olddir /home2/apache/logs/oldlogs postrotate /usr/bin/killall -HUP httpd endscript } /home2/apache/logs/*-combref_log { rotate 9 monthly errors [email protected] create ifempty olddir /home2/apache/logs/oldlogs postrotate /usr/bin/killall -HUP httpd endscript } /home2/apache/logs/*-error_log { rotate 9 monthly errors [email protected] create ifempty olddir /home2/apache/logs/oldlogs postrotate /usr/bin/killall -HUP httpd endscript }For a log file of an ad-hoc utility I wrote, I have now created this configuration file /etc/logrotate.d/dping for `logrotate'.
# The new improved logrotate script for dping on fox. /var/log/dping.txt { rotate 9 monthly compress # delaycompress copytruncate ifempty errors [email protected] postrotate endscript }Since the creating process cannot be HUPped to make it work with a new log file, I've used `copytruncate' to copy the file and then truncate it. The `delaycompress' is not useful because the original log file is copied rather than moved. Creation of .cvsignore fileNow on Sunday 30 September 2001, it's time to get `logrotate' going on my new SuSE 7.1 web server `dog'. The problem is that SuSE 7.1 does not come with logrotate software on it. So I had to go and look for it. The link at redhat was wrong, but by looking around a bit, I finally found it at ftp://ftp.redhat.com/pub/redhat/linux/code/logrotate/. I downloaded this file: logrotate-3.3.tar.gz.
[2005-5-5: I think it has disappeared again now. But here's a backup copy of logrotate-3.3.tar.gz.]
This can be compiled very simply with `make' and `make install'. There's a manual logrotate.8 with it too. The binary is installed as /usr/sbin/logrotate. So now all I have to do is write a configuration file for the logrotate program and then write a cron script for it.(Note that since my main initial consideration is to rotate the httpd logs for the beginning of October 2001, I could just use the Apache tool for this purpose, which is /usr/local/apache/bin/rotatelogs on my system. However, I can't understand the documentation for this. So I'm playing safe and using the more flexible redhat `logrotate' tool instead.)
On my SuSE 7.1 machine `dog', the manual says that after reading the per-user crontab files in /var/spool/cron/tabs on start-up, the cron process reads the file /etc/crontab. On my machine as configured, I find the following.
root@dog# more /etc/crontab SHELL=/bin/sh PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin MAILTO=root #-* * * * * root test -x /usr/sbin/atrun && /usr/sbin/atrun 0 21 * * * root test -x /usr/sbin/faxqclean && /usr/sbin/faxqclean 5 22 * * * root test -x /usr/sbin/texpire && /usr/sbin/texpire 25 23 * * * root test -e /usr/sbin/faxcron && sh /usr/sbin/faxcron | mail FaxMaster # # check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly # -*/15 * * * * root test -x /usr/lib/cron/run-crons && /usr/lib/cron/run-crons 59 * * * * root rm -f /var/spool/cron/lastrun/cron.hourly 14 0 * * * root rm -f /var/spool/cron/lastrun/cron.daily 29 0 * * 6 root rm -f /var/spool/cron/lastrun/cron.weekly 44 0 1 * * root rm -f /var/spool/cron/lastrun/cron.monthlyThere's nothing here to help me with initiating my daily script. (By the way, the fax commands are a bit worrying. I'll get rid of those when I understand exactly what they do. They obviously produce many meaningless message which root receives every day!)So my next step is to look at the files in /etc/cron.d, because the cron manual says that all scripts in this directory are read next. On my machine the only file in /etc/cron.d is a script `seccheck', which produces copious useless messages to root every day and week. (I'll see if I can get rid of that some day too!)
The directory /etc/cron.daily contains a script `aaa_base_rotate_logs' which contains a complex set of rotation rules, but how are the scripts in this directory invoked? Hmmm... Maybe they're invoked from that `/usr/lib/cron/run-crons' script. Yes!! That's where it's invoked from. Yet another big, incomprehensible script. The core of that script is the following Bourne-shell loop.
SPOOL=/var/spool/cron/lastrun for CRONDIR in /etc/cron.{hourly,daily,weekly,monthly} ; do test -d $CRONDIR || continue BASE=${CRONDIR##*/} test -e $SPOOL/$BASE && { case $BASE in cron.hourly) TIME="-cmin +60 -or -cmin 60" ;; cron.daily) TIME="-ctime +1 -or -ctime 1" ;; cron.weekly) TIME="-ctime +7 -or -ctime 7" ;; cron.monthly) TIME="-ctime +`date -u +%d`" ;; esac eval find $SPOOL/$BASE $TIME | xargs -r rm -f } if test ! -e $SPOOL/$BASE ; then touch $SPOOL/$BASE # keep going when something fails set +e for SCRIPT in $CRONDIR/*[^~,] ; do test -d $SCRIPT && continue test -x $SCRIPT || continue case "$SCRIPT" in *.rpm*) continue ;; *.swap) continue ;; *.bak) continue ;; *.orig) continue ;; \#*) continue ;; esac /sbin/checkproc $SCRIPT && continue nice -15 $SCRIPT done fi doneNow what does this mean?
The command `BASE=${CRONDIR##*/}' means that BASE is set to the `longest substring of $CRONDIR which matches pattern "*/"', according to my Bash reference card. This just means that the leading path components are removed. (This is done more simply in C-shell!) In the case of the daily cron job, if there is a file /var/spool/cron/lastrun/cron.daily (which is true), then the following command is run.eval find /var/spool/cron/lastrun/cron.daily -ctime +1 -or -ctime 1 | xargs -r rm -fThe `xargs' command (which I have never seen before) builds and executes a command line from standard input. Weird! The `xargs' manual says this.If the standard input does not contain any non-blanks, do not run the command. Normally, the command is run once even if there is no input.So in this case, the command `rm -f' is executed for each of the files with modification times within 24 hours of the current time. I don't really understand this.It looks like the file removal commands in the file /etc/crontab are designed to synchronise the operation of the quarter-hour operations. The commands are only executed if the files in /var/spool/cron/lastrun have been removed. What a convoluted way of achieving a simple objective!!
Next any file in the directory /etc/cron.daily which do not end in the characters `~' or `,' (presumed to be edited files) are executed if they are executable and do not have the endings .rpm, .swap, .bak or .orig as follows.
/sbin/checkproc $SCRIPT && continue nice -15 $SCRIPTThe loop continues is the process is already running. Otherwise it is run with nice level 15.What this all means finally is that any script in the directory /etc/cron.daily will be run at 00:00 on each day. More to the point, the /etc/cron.monthly scripts are run at 00:00 at the beginning of each month. It all looks a bit dodgy because the `date' test is `date -u', which gives the current UTC day. But this should work, although the motivation for the `-u' is not quite clear.
All I have to do now is write a script and put it in /etc/cron.monthly. No problem!
Sys Admin
After a system failure, tracking down the media, determining the last known good backup, and starting the recovery process can be tricky. I implemented the solution described in this article to reduce the stress of performing backups. Using a program found in Linux called logrotate, I created a backup solution to consolidate and automate the backup process. Logrotate's configuration file contains the details to create the backups, compress the backup files, provide automatic rotation, and remove stale files.
Logrotate is a utility designed to handle the management of log files. Logrotate maintains a series of log files, automatically rotates the files, and removes the files based on the parameters provided in a configuration file. You can specify the name of any file in the logrotate configuration file, it is treated the same as a log file.
I used logrotate when I worked with Red Hat (RH) 6.0. Since then, I have found it to be part of a standard RH install. The current version is available from:
http://www.rpmfind.net/linux/RPM/The program can be installed with an rpm -Uvh logrotate-3.6.4-1.i386.rpm (current RH 7.3 version) command. Following are a few points of interest in the logrotate tour:/usr/sbin/logrotate -- Utility residence
/etc/logrotate.conf -- Default configuration file
/etc/logrotate.d -- Default directory for other configuration files
/var/lib/logrotate.status -- Default state file
On RH, logrotate is run from a daily job directory /etc/cron.daily/logrotate. When logrotate is called, it obtains information about which files it is processing by using the default configuration file /etc/logrotate.conf. Alternate configuration files can be used if optioned. The configuration file defines the file to rotate and the parameters to apply. A configuration file entry may look like this:
#daily "/var/log/logfile.log" { daily rotate 7 compress }Using the configuration shown here, the file logfile.log would be compressed and rotated and given the name of logfile.log.1.gz. After day two, the file logfile.log.1.gz would be rotated to logfile.log.2.gz and new copy of logfile.log.1.gz produced from logfile.log. At the end of the configuration file schedule, the directory would contain seven copies of the database. On the eighth day, the copy with the extension logfile.log.7.gz would be replaced with another rotated copy. A long listing (ls -al) in the directory would display seven copies of the database files, with dates preserved from when they were first created.The man pages for logrotate explain the parameters used to control logrotate. The remainder of this article describes how logrotate can be used to produce a backup schema.
When planning the backup process, I used a variation on the grandfather-father-son backup tape rotation method. The backup set consists of grandfather(monthly), father(weekly), and son(daily) as shown in Figure 1.
Under this grandfather-father-son implementation method, a full backup is performed every day (Monday-Sunday); a full backup of the oldest daily is preserved every Sunday; a full backup of the oldest weekly is preserved the last day of a four-week cycle. This assumes you start the daily cycle on a Monday. After 1 year, 6 dailies, 3 weeklies, and 12 monthly copies are preserved.
Logrotate works similarly to a stack in programming. As each file is rotated, it is pushed onto the stack and forces other files down the stack. The size of the stack is determined by the rotate value in the logrotate configuration file. Once the stack capacity is reached, the first file in gets pushed out. Figure 1 illustrates the idea.
To produce the backup schema shown in Figure 1, create the configuration file /etc/logrotate_backup.conf from the following listing:
############################## # #Name:logrotate_backup.conf # #Description:script designed to produce backups #grandfather(monthly),father(weekly),son(daily) # #Author: Sean D. Conway # #Wed Aug 28 18:12:59 CDT 2002 # #Revision: # #NOTE: #replace the path to be backed up. #don't remove "" good for files that have spaces. #maintain the "." file extensions this will preserve the frequency ################################ #daily (son) "/home/oracle/backup/database" { daily rotate 7 compress } #weekly (father) "/home/oracle/backup/database.7.gz" { weekly rotate 4 } #monthly (grandfather) "/home/oracle/backup/database.7.gz.4" { monthly rotate 12 }This configuration file is divided into three sections: daily, weekly, and monthly. The following listing provides some comments to explain the parameter lines found in the script:#daily (son) <--comments line "/home/oracle/backup/database" { <--file to rotate daily <--rotation schedule rotate 7 <--rotation interval compress <--compress the file }The parameters here enable the logrotate program to rotate and compress daily a file called database. The rotation interval is seven. At the end of one week, the storage directory will contain the following files: database.1.gz, database.2.gz, database.3.gz, database.4.gz, database.5.gz, and database.6.gz. The weekly configuration would rotate the file database.7.gz to database.7.gz.1. The file database.1.gz would be the most current.The logrotate utility can access the configuration files using two methods:
1. The configuration file can be placed in the default configuration file directory /etc/logrotate.d. This directory is listed in logrotates default configuration file /etc/logrotate.conf. All files in /etc/logrotate.d are additional configuration files that will be processed by the program.
2. An alternate method is to force the logrotate program to use an alternate configuration file from a command line. The following is the cron entry to support logrotate running each day at 07:00 using a configuration file called /etc/logrotate_backup.conf.
00 07 * * * /usr/sbin/logrotate -f /etc/logrotate_backup.confI prefer the second method because it allows more flexible scheduling.The logrotate backup mechanism is used to support three database servers as shown in Figure 2. Each database server produces a backup file and sends it to a server called a tape_gate. On the tape_gate server, logrotate runs daily and takes care of producing the rotation schema. The complete backup series is exported to tape for offsite storage.
Tape_gate also runs a licensed tape client for accessing the tape silo. This configuration allows the administrators to use the client software to schedule and access the backups. The administrator can use the backups stored locally or pull the file back from the tape storage device. All the file creation dates are preserved, and a simple grep on a month name will produce a list of backup files available for that month.
To get support for this backup solution, I had to address some design limitations. This implementation requires a server to process the backups. The horsepower required for the server is minimal. For this installation, I used a reclaimed Pentium desktop with an additional cost of two hard drives. It does provide an alternate point for accessing backups. I find it easier to access a backup from a disk rather than searching a tape library.
Backup files are moved from the source to tape_gate using secure ftp. At selective network slow periods the additional network traffic is acceptable. Secure ftp provides access to a ready backup from tape_gate if needed.
Table 1 shows the complete set of filenames after a one-year cycle. The drive capacity must be sufficient to support the number of system backups being handled. The cost for hard drive capacity today is lower than it was ten years ago. Sixty-gig IDE drives are common. At 14-MB compressed, I was able to get all the backups on a 60-GB drive. The three-database server backups total 500 MB compressed.
The feature that sold this design was cost. A client license for accessing the tape silo costs $1200. Each of the databases runs a different operating system, so each would have needed a different client license. Instead, two drives cost $300 and $1200 for one client license for tape_gate; I saved $2100. The alternative was to create NFS mount on all the servers.
I have been asked to add additional backups to the server. It seems this common approach to backups for a variety of systems can make life easier for administrators. Certainly, one common backup mechanism for all systems is easier to remember than a different backup method for each system.
Sean D. Conway is a former college instructor turned Network System Specialist for a regional telecommunications company in Canada. Working with administrators, they are responsible for the care and feeding of operational support systems inside the telecommunication cloud. He still dabbles in network theory lectures. His formal electronic engineering technology training and education background are valuable tools in achieving a goal -- making learning about computers and networks easier.
Google matched content |
logrotate is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large.
Normally, logrotate is run as a daily cron job. It will not modify a log multiple times in one day unless the criterium for that log is based on the log's size and logrotate is being run multiple times each day, or unless the -f or -force option is used.
Any number of config files may be given on the command line. Later config files may override the options given in earlier files, so the order in which the logrotate config files are listed is important. Normally, a single config file which includes any other config files which are needed should be used. See below for more information on how to use the include directive to accomplish this. If a directory is given on the command line, every file in that directory is used as a config file.
If no command line arguments are given, logrotate will print version and copyright information, along with a short usage summary. If any errors occur while rotating logs, logrotate will exit with non-zero status.
Turns on debug mode and implies -v. In debug mode, no changes will be made to the logs or to the logrotate state file.
logrotate reads everything about the log files it should be handling from the series of configuration files specified on the command line. Each configuration file can set global options (local definitions override global ones, and later definitions override earlier ones) and specify logfiles to rotate. A simple configuration file looks like this:
# sample logrotate configuration file
compress
/var/log/messages {
rotate 5
weekly
postrotate
/usr/bin/killall -HUP syslogd
endscript
}
"/var/log/httpd/access.log" /var/log/httpd/error.log {
rotate 5
mail [email protected]
size 100k
sharedscripts
postrotate
/usr/bin/killall -HUP httpd
endscript
}
/var/log/news/* {
monthly
rotate 2
olddir /var/log/news/old
missingok
postrotate
kill -HUP 'cat /var/run/inn.pid'
endscript
nocompress
}
The first few lines set global options; in the example, logs are compressed after they are rotated.
Note that comments may appear anywhere in the config file as long as the first non-whitespace character
on the line is a #.
The next section of the config files defined how to handle the log file /var/log/messages. The log will go through five weekly rotations before being removed. After the log file has been rotated (but before the old version of the log has been compressed), the command /sbin/killall -HUP syslogd will be executed.
The next section defines the parameters for both /var/log/httpd/access.log and /var/log/httpd/error.log. They are rotated whenever it grows over 100k in size, and the old logs files are mailed (uncompressed) to [email protected] after going through 5 rotations, rather than being removed. The sharedscripts means that the postrotate script will only be run once (after the old logs have been compressed), not once for each log which is rotated. Note that the double quotes around the first filename at the beginning of this section allows logrotate to rotate logs with spaces in the name. Normal shell quoting rules apply, with ', ", and \ characters supported.
The last section defines the parameters for all of the files in /var/log/news. Each file is rotated on a monthly basis. This is considered a single rotation directive and if errors occur for more than one file, the log files are not compressed.
Please use wildcards with caution. If you specify *, logrotate will rotate all files, including previously rotated ones. A way around this is to use the olddir directive or a more exact wildcard (such as *.log).
Here is more information on the directives which may be included in a logrotate configuration file:
Make a copy of the log file, but don't change the original at all. This option can be used, for instance, to make a snapshot of the current log file, or when some other utility needs to truncate or parse the file. When this option is used, the create option will have no effect, as the old log file stays in place.
Log files are rotated every day.
Do not copy the original log file and leave it in place. (this overrides the copy option).
Don't mail old log files to any address.
Delete log files using shred -u instead of unlink(). This should ensure that logs are not readable after their scheduled deletion; this is off by default. See also noshred.
Log files are rotated if the current weekday is less than the weekday of the last rotation or if more than a week has passed since the last rotation. This is normally the same as rotating logs on the first day of the week, but it works better if logrotate is not run every night.
yearly
Log files are rotated if the current year is not the same as the last rotation.
Default state file.
/etc/logrotate.conf
Society
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
Quotes
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Bulletin:
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
History:
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
Classic books:
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D
Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|
You can use PayPal to to buy a cup of coffee for authors of this site |
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.
Last modified: July 28, 2019