Syslog Configuration and /etc/syslog.conf

• Introduction
• Importance of syslog analysers
•  /etc/syslog.conf  file, selectors and action field
• The rule "this level and higher" for selectors
• Classic /etc/syslog.conf  file
  • Selector and action fields
• Creating your own custom log files
• Implementations
• Standard syslog components
• Syncronious vs asynchronous  writes
• Sources of log messages
• Targets of syslog: writing log to something other then a log file
• Troubleshooting
• What's wrong with syslog
• Links to additional information

Introduction

One of the most neglected area of Unix is handling system events. Daily check for unusual system messages is crucial for the security and health conditions of a computer system. Syslog daemon was created as "afterthought" and is actually a result of Sendmail development, not Unix kernel development. Initial implementation was way too primitive and inflexible.  The rotation of log files in not a function of standard syslog and is performed by a special program. In Linux this is logrotate

Believe it or not, but syslog was create as "afterthought".  It was initially created for sendmail and became part of Unix only later. Due to this some architectural decisions made in syslog are suboptimal. Still attempt to improve syslog such as syslog-ng and rsyslog proved to be only half-successful. The inertia of already established, critical,  even if highly deficient subsystem proved to be tremendous.

The fact that syslog daemon mutated out of a Sendmail debugging aid into the standard log file daemon  for Unix has good and bad consequences.  A lot of daemons and applications now send their log output to syslog so it became standard de-facto. But format of the log message is restrictive and facilities (nothing more than "categories" of log messages) and severities got introduced in a very limited, not well though out, form. The actual filtering/distribution of messages to log files is defined in syslog /etc/syslog.conf file.  Format of this file and flexibility of creation own classes of messages can also be improved.

Two recent implementations of syslog daemon (rsyslog and syslog-ng) managed somehow to lessen the severity of those shortcomings.  They have different functionaly and generally can be used both for different purposes:

• Since version 5 rsyslog is default in RHEL and  syslog-ng in SLES (since SLES 10). You can install rsyslog instead of syslog-ng in SLES 11.
• As syslog-ng is an overkill for individual server and has most value for central loghost server

It its pure form, classic syslogd provides an outdated, insecure (unless it is used with the central logging host, as it actually should be used in any modern enterprise environment),  but simple and very reliable logging mechanism. It definitely lacks the flexibility of dynamically extending message classification scheme, it has almost no built-in security, but it does the job.  

A better message classification scheme was proposed in 2004 syslog-1 but tremendous amount of inertia prevents any meaningful changes.

Importance of syslog analysers

Logs of a modern Unix system typically contain much "noise" -- messages which have no importance. But those "useless" or "spam" messages are so numerous that few important events are easily lost in the volume of messages. that create a need for syslog analyzers, which evolved into an important class of utilities.

With typical regular expression based analyzers like Logwatch (or its, less primitive, but similarly architecturally deficient commercial counterparts like  Splunk) it's difficult to select which messages we are interested in, as generally you are limited only to past events "patterns".

Here anomaly detection analyzers can probably be more useful. 

 /etc/syslog.conf  file, selectors and action field

The key file that is influencing syslog behavior is /etc/syslog.conf  file.  It consist of set of rules, each of which has two parts: set of selectors (semicolon delimited) and set of actions (comma delimited; space after comma is allowed). In other words each line of the /etc/syslog.conf file contains two parts:

• List of selectors that specifies which kinds of messages to log (e.g., all error messages or all debugging messages from the kernel).
• Single action that says what should be done with the message (e.g., put it in a file or send the message to a user's terminal).

The basic syntax of /etc/syslog.conf  is easy

1. Empty lines and everything behind a hash mark (#) is ignored
2. Rules are of the format

 <selector>;[<selector]...    <Actions>

Each selector is a pair that combines a facility and a severity separated by a period:

  <facility>.<severity>

Severity in selector means range from given severity to the top severity. That means that such selector will catch all messages belonging to the given facility that have the given severity and higher.

Now let's discuss those two columns is some details. The selector list is a semicolon-separated list of priority specifications in the following format: facility.level; facility.level.

Traditionally it contains two columns called the selection and action, respectively. 

• selector field: a semicolon-separated list of priority specifications consisting of pairs in the following format: facility.level; facility.level...  For example

  err;kern.debug;daemon.notice;mail.crit

• action field:  defines where to forward the message. This field can have one or several of the following blank or newline separated entries:
  1. file  -- Output sent to the file specified
  2. @host -- The @ sign denotes that messages must be forwarded to a host  The name should be defined in /etc/hosts and represent a remote logserver.
  3. user[,user] -- The user(s)  (can be comma delimitated list) receive messages if they are logged in.
  4. * -- All logged in users will receive messages when they are logged in.

In classic syslog daemon a message can be selected based on the its  facility/priority pair(selector).

• There are 12+8 (12 real and 8 local) predefined facilities (kernel, mail,  daemon, news, syslog, lpr, auth, uucp, cron, local1-8, mark etc.).  See Syslog Messages Classification. Fixed facilities is a big shortcoming of standard syslog (a design blunder) . Some of them are no longer used (uucp). Fixed facilities is the major weakness of classic syslog

• There are eight different priorities (EMERG, ALERT,  CRIT,  ERR,  WARN,  NOTICE, INFO, DEBUG). Fixed priorities have a strange order and some of them (like ALERT and NOTICE)  have badly chosen names. For example, in first three (most severe) priorities (EMERG, ALERT,  CRIT), why on the earth ALERT has higher priority then CRIT messages ?  What is funny that older IBM mainframe errors severity classification (from early 60th) is much better (talk about the progress after that ;-). Still even in its present form it is an OK solution.

If you only want to catch messages belonging to exactly the given severity, prefix the priority with an equation sign (=):

 <facility>.=<severity>

You can also negate the severity selection by prepending an exclamation sign (!):

 <facility>.!<severity>

This will select all messages belonging to the given facility and that have a severity lower than the one specified. Note that this also weeds out messages belonging to the given severity - which is logical, since the opposite of >= is <.

Of course this can make things tedious if you have to list all combinations of the 20 facilities and 9 severities by hand. So there are shortcuts, such as specifying an asterisk (*) as a catchall:

  <facility>.*  -> All messages belonging to <facility> 
  *.<severity>  -> All messages of the given <severity> 
  *.*                 -> All messages

And then, you can specify lists of "whats", where the "whats" are delimited by semicola (;):

 <facility>.<severity>;<facility>.<severity>

Or, if you want to process the same severities of different facilities, list the facilities using commas (,) first:

 <facility>,<facility>.<severity>

To make matters interesting, there is also a special severity called "none", which implies that no message of the given facility are to be logged with this rule:

  *.*;<facility>.none  -> Log all messages except those of the given facility

The rule "this level and higher" for severity level in selectors

Let's remind the basic structure of the syslog.conf.

•  The facility field can contain only 17 codes:
  • kern Messages generated by the kernel.
  • user Messages generated by user processes.
  • mail The mail system.
  • daemon System daemons, such as the in.ftpd and the telnetd daemons.
  • auth The authorization system, including the login and su commands.
  • syslog Messages generated internally by the syslogd daemon.
  • lpr The line printer spooling system, such as the lpr and lpc commands.
  • news Files reserved for the USENET network news system.
  • uucp  (obsolete) The UNIX-to-UNIX copy (UUCP) system does not use the syslog function.
  • cron The cron and at facilities, including crontab, at, and cron.
  • mark  Timing  messages. For example mark.* /dev/console line causes the time to be printed on the system console every 20 minutes. This is useful if you have other information being printed on the console, and you want a running clock on the printout.
  • local0-7 Eight user-defined codes.
• the level selector specifies the severity or importance of the message. Each level includes all the levels above (of a higher severity).  To remember the sequence for the certification exam you can use an appropriately constructed phase like "Every alerted cardriver escapes warning notice"

  • emerg or 0 Panic conditions that are normally broadcast to all users

  • alert or 1 Conditions that should be corrected immediately, such as a corrupted system database. Only sysadmin of a particular server needs to be informed by mail or paged.

  • crit or 2 Warnings about critical conditions, such as hard device errors. 

  • err  or 3 Errors other than hard device errors

  • warning  or 4 Warning messages, that generally does not interfere with normal operation.

  • notice or 5 Non-error conditions that might require special handling

  • info  or 6 Purely informational messages (usually does not require any handling)

  • debug or 7 Messages that are normally used only when debugging a program

  • none  or 8 Messages are not sent from the indicated facility to the selected file

Any severity level in selector of /etc/syslog.conf means "this level and higher", for example mail.crit includes mail.alert and  mail.emerg)

Action fields

The action field specifies single action that needs to be done with selected log message.  In some implementations actions can be combined (using comma with optional space as separators), for example:

auth.notice                      /var/adm/authlog, /var/log/messages

also, in Solaris when syslogd daemon starts it does not read /etc/syslog.conf file directly. Instead is starts m4 microprocessor, which parses the /etc/syslog.conf file and processes all ifdef statements (and only them). The function ifdef allow to check presence of system identified as LOGHOST. The macro then can evaluate if log file are to be help locally or on a remote system, or a combination of both.

There are  five possible actions (some versions of syslog support additional actions, such as logging to a proprietary error management system):

1. Log to a file or a device In this case, the action field consists of a filename (or device name), which must start with a forward slash (e.g., /var/adm/lpd-errs or /dev/console). Beware: logging to /dev/console creates the possibility of a denial of service attack. If you are logging to the console, an attacker can flood your console with log messages, rendering it unusable. If your system supports virtual consoles, as with Linux, you can usually safely log to one of the virtual consoles, and leave the others uncluttered.
    
2. Send a message to a user. In this case, the action field consists of a username (e.g., root). You can specify multiple usernames by separating them with commas (e.g., root,secadmin). The message is written to each terminal where these users are shown to be logged in, according to the utmp file.
    
3. Send a message to all users. In this case, the action field consists of an asterisk (*).
    
4. Pipe the message to a program. In this case, the program is specified after the Unix pipe symbol (|). Note that the current Solaris version syslog does not support logging to programs. You need to use syslog-ng to achieve that.
    
5. Send the message to the syslog on another host. In this case, the action field consists of a hostname preceded by an at sign (e.g., @LOGHOST).

IMPORTANT REMINDER: In classic syslogd  the syntax of /etc/syslog.conf is pretty restrictive. It does not permit using spaces as a separator, tabs should be used as a separator between list of selectors and list of actions.

Let's discuss some examples:

*.err;kern.debug;auth.notice /dev/console

This line causes all error messages, all kernel debug messages, and all notice messages generated by the authorization system to be printed on the system console. If your system console is a printing terminal, this process will generate a permanent hardcopy that you can file and use for later reference. (Note that kern.debug means all messages of priority debug and above.)

daemon,auth.notice /var/log/messages

This line causes all notice messages from either the system daemons or the authorization system to be appended to the file /var/log/messages. Note that this is the second line that mentions auth.notice messages. As a result, auth.notice messages will be sent to both the console and the messages file.

lpr.err /var/log/lpd-errs 

This line causes all messages with severity err and above from the line printer system to be appended to the /var/log/lpd-errs file.

auth.* root,secadmin  

This line causes all messages from the authorization system to be sent to the users root and secadmin. Note, however, that if the users are not logged in, the messages will be lost.

auth.* @LOGHOST

This line causes all authorization messages to be sent to the syslog daemon on the computer defined in /etc/hosts file as LOGHOST.

*.emerg *

This line causes all emergency messages to be displayed on every user's terminal.

*.alert | logpipe

This line causes all alert messages to be sent to a program called logpipe, which might forward them to some kind of monitoring system for display.

mark.* /dev/console

This line causes the time to be printed on the system console every 20 minutes. This is useful if you have other information being printed on the console, and you want a running clock on the printout.

Making changes to syslog.conf file

After making any changes to syslog.conf file, you need to ask the daemon to reread the configuration file with kill -HUP command, for example pkill -HUP syslogd.

This is an operation that is often forgotten. It might make sense to implement "system configuration" attribute that can automatically send executes a command after closing of the file with such attribute if it was opened for writing (Unix has "command execution string" for scripts for example #!/usr/bin/perl, so similar notaion which specify the "interpreter" can be used for configuration files).

In the absence of such facility, which would be a real paradise for absent minded people like me, you probably will be better off creating a special script, like visyslog  that contains two command: vi and pkill to ensure that you do not forget this operation; I often do and then face consequences):

[ -e "$1."`date +"%y%m%d"`".bak" ] && cp -p $1 $1.`date +"%y%m%d"`.bak
vi $1
pkill -HUP syslogd

Creating your own custom log files

See also Syslog Configuration Examples. It contain eight examples of syslog.conf connected on Internet.

One of the most typical weaknesses of a typical enterprise Unix/Linux server installation is usage of default syslog configuration file. It just does not make sense to use default syslog configuration file in any environment as a lot of messages in it are junk.  It always should be customized. That's number one rule for any system administrator who respects his time.

It is important to remember that for any severity level specified, messages are selected for this is more severe error types. For example, kern.warning causes warning-, err-, crit-, alert-, and emerg-level messages from the kernel service to be written to particular log.

The default Solaris syslog configuration (/etc/syslog.conf) is far from being optimal (again, any selector in /etc/syslog.conf means "this severity level and higher", for example mail.crit includes mail.emerg):

*.err;kern.notice;auth.notice                 /dev/sysmsg
*.err;kern.debug;daemon.notice;mail.crit      /var/adm/messages

*.alert;kern.err;daemon.err                   operator
*.alert                                       root
*.emerg                                       *

# if a non-loghost machine chooses to have authentication messages
# sent to the loghost machine, un-comment out the following line:
#auth.notice ifdef('LOGHOST', /var/log/authlog, @loghost)

mail.debug ifdef('LOGHOST', /var/log/syslog, @loghost)

#
# non-loghost machines will use the following lines to cause "user"
# log messages to be logged locally.
#
ifdef(`LOGHOST', ,
user.err                                     /dev/sysmsg
user.err                                     /var/adm/messages
user.alert                                   root, operator
user.emerg                                     *
)

For one thing, in Solaris AUTH messages by default don’t get logged to any logfiles. This is important if you want to know when people are trying to break into your system so such messages should be emailed at least to operator (may be operator and root) and written to /var/adm/authlog

*.emerg                          *
*.kernel.notice;*.alert          root, operator
*.err;kern.notice;auth.notice    /dev/sysmsg
*.notice                         /var/adm/messages 
auth.notice                      /var/adm/authlog, /var/log/messages

You can select messages from the daemon that interests you and send it to a separate file, That save time and effort of grepping messages file all the time

For example, kernel.err, *.alert and *.emerg deserve a separate file

kernel.err;*.alert;*.emerg          /var/adm/errors

The syslog.conf below is from Kai O'Yang who forwarded his syslog.conf files to share.

Here's my syslog.conf for all client machines.

#ident  "@(#)syslog.conf        1.3     93/12/09 SMI"   /* SunOS 5.0 */ #
# Copyright (c) 1991-1993, by Sun Microsystems, Inc. #
# syslog configuration file.
#
# This file is processed by m4 so be careful to quote (`') names # 
that match m4 reserved words.  Also, within ifdef's, arguments # 
containing commas must be quoted.
#
# Note: Have to exclude user from most lines so that user.alert #      
and user.emerg are not included, because old sendmails #       will 
generate them for debugging information.  If you
#       have no 4.2BSD based systems doing network logging, you #      
can remove all the special cases for "user" logging.
#
*.err;kern.notice;auth.notice;user.none        	 /dev/console 
*.err;kern.debug;daemon.notice;mail.crit;user.none  /var/adm/messages @loghost

*.alert;kern.err;daemon.err;user.none           operator,@loghost 
*.alert;user.none                               root,@loghost

*.emerg;user.none                               @loghost

auth.info;mail.info;daemon.info       		@loghost

#ident  "@(#)syslog.conf        1.3     93/12/09 SMI"   /* SunOS 5.0 
*/ #
# Copyright (c) 1991-1993, by Sun Microsystems, Inc. #
# syslog configuration file.
#
# This file is processed by m4 so be careful to quote (`') names # 
that match m4 reserved words.  Also, within ifdef's, arguments # 
containing commas must be quoted.
#
# Note: Have to exclude user from most lines so that user.alert #      
and user.emerg are not included, because old sendmails #       will 
generate them for debugging information.  If you
#       have no 4.2BSD based systems doing network logging, you #      
can remove all the special cases for "user" logging.
#
*.err;kern.notice;auth.notice;user.none         /dev/console 
*.err;kern.debug;daemon.notice;mail.crit;user.none     /var/adm/messages

*.alert;kern.err;daemon.err;user.none           operator 
*.alert;user.none                               root

*.emerg;user.none                               * auth.info            
        					/var/log/authlog mail.info
				/var/log/mlog
#
# Adding log to daemon information
#
daemon.info     /var/log/syslog

Implementations

There are multiple implementations of syslog. With some attempting to rectify shortcomings of the original (classic) syslog daemon:

• Standard syslog (still used in Solaris)

• RHEL uses rsyslog since version 5.1. Versions before that used GNU clone of classic syslogd. It is now a drop-in replacement for classic syslogd. It also can be installed in Suse instead of syslog-ng. The structure of rsyslog files is similar as it is "drop-in" replacement for classic syslogd. In RHEL 6.5 version 5.8.10 of rsyslogd is used.

• SLES 9-11 uses syslog-ng by default,  It has completely different configuration files. The latest version (SLES 11 SP3) can use rsyslog instead (DVD provides RPM package and /etc/init.d/syslog script checks if it installed and if so uses it instead of syslog-ng. It looks like SLES is moving toward replacement of syslog-ng with rsyslog. Older versions use classic syslogd clone. Overcomplexity prevents it wide usage and it is the most attractive as log daemon for central log server, loghost (see Remote Syslog )

Classic syslog does not have built-in log rotations capabilities. Linux used logrotate  utility for log rotation (both in RHEL and SLES).  See Linux logrotate

Standard syslog components

Standard Solaris syslog is still "classic System V syslogd" and consists of the following components:

• syslogd — the system daemon used to receive and route system log events from syslog() calls and logger commands

• /etc/syslog.conf — the configuration file used to control the logging and routing of system log events

• logger — a UNIX command used to add single-line entries to the system log. See also ogger

• syslog() — an application program interface (API) referenced by several standard system utilities and available to anyone writing software in the C programming language, C++, Java and many scripting languages (Perl is one example).

• Log rotation utility
  • logrotate -- standard utility for rotation of logs in linux
  • logadm -- System Log Rotation Utility (Solaris 9 and later)

Syncronious vs asynchronous  writes

After the "selectors" part with all their twists and turns, the action part of rules looks pretty simple:

 </path/to/logfile>

will log everything to the given logfile.  But in reality it is not that simple, and have its own set of twists and turns.

Logging is done with synchronous writes, which means that after each log entry, syslog waits for the operating system kernel to acknowledge that the data has indeed been written to the disk before writing its next entry. This can slow down your system 10-fold for services with extensive logging (especially corporate mail servers!). This effect has been verified, so for servers/applications that generate tremendous amount of log files, if you can afford to write logs asynchronously, just do it. Reserve synchronous writes only to top priority messages (see example of error log definition above):

kernel.err;*.alert;*.emerg          /var/adm/errors

To indicate to syslog that you want log entries to be written asynchronously, prepend a minus (-) to the logfile:

 -</path/to/logfile>

You should already use this option for mail and http logs. For example

mail.*                           -/var/log/maillog

This is basically what is needed in 99% of syslog messages with low priorities. 

Note that you can specify the multiple targets as well as multiple selectors.  The messages will then be logged to all targets specified.

Sources of log messages

Syslog generally can receive messages in three ways:

• Through the syslog() function most languages provide (after an appropriate call to openlog())
• Through named sockets such as /dev/log which is enabled by default on most distributions
• Via UDP on port 514, if syslogd is running with the -r option.
  • Note that there is no authentication or authorization implemented in the standard syslog protocol! Caveat emptor!)

One important feature of SYSLOG is the ability send messages via UDP on port 514 and then aggregate messages sent from multiple servers on a special server. Special BSD syslog protocol defined in RFC 3164  is used. It uses target UDP port 514.  RFC recommends that source port also be set to 514. A good summary of RFC can be found in Introduction to Syslog Protocol - MonitorWare

syslog uses the user datagram protocol (UDP) [1] as its underlying transport layer mechanism.  The UDP port that has been assigned to syslog is 514.

It is RECOMMENDED that the source port also be 514 to indicate that the message is from the syslog process of the sender,  but there have been cases seen where valid syslog messages have come  from a sender with a source port other than 514.  If the sender uses  a source port other than 514 then it is RECOMMENDED and has been  considered to be good form that subsequent messages are from a single
consistent port.

Targets of syslog: writing log to something other then a log file

You have at least five other choices:

• Remote machines:

   @<hostname>

• Named Pipes:

   |<path to fifo>

• Terminals by giving their device files as logfiles
• Specific users (if they're logged on) using write:

   <user>,<user>

• All users logged on:

   * 

These are things you don't need that often, and if you do, you'd better read up all the information available on them in the manpage first!

Troubleshooting

See also Syslog configuration debugging

Overall, the suggestions are just common sense and most of them were mentioned before: 

• Don't use spaces, use tabs when configuring syslog.conf.
• After making changes send HUP sygnal to syslog, kill -HUP pid for syslog.conf.
• Add a loghost alias in /etc/hosts or NIS table for the syslog host.  This host should not be dependent of DNS resolution.

syslogd_flags="-d -a logclien.example.com -v -v"

and issue a restart:

service syslogd restart

What's wrong with syslog

Here is an interesting discussion from Rainer blog: Serious syslog problems?

In the paper introducing journald/Linux Journal a number of shortcomings in current syslog practice are mentioned. The authors say:

Syslog has been around for ~30 years, due to its simplicity and ubiquitousness it is an invaluable tool for administrators. However, the number of limitations are substantial, and over time they have started to be serious problems:

I have now taken some time to look at each of these claims in depth. But before I start, I need to tell that I am working in the IT logging field for nearly 15 years, have participated in a number of standards efforts and written a lot of syslog-related software with rsyslog being a prime example (some commercial tools I have been involved with can be found here). So probably I have a bias and my words need to be taken with a grain of salt. On the other hand, the journald authors also have a bias, so I guess that's a fair exchange of arguments ;).

In my analysis, I compare the journald effort with what rsyslog currently provides and leave closed source software out. It is also important to note that there is a difference between syslog, the protocol, a specific syslog application (like rsyslog) and a system log message store. Due to tradition, these terms are often used for different things and one must deduce from context, what is meant. The paper applies the same sloppiness in regard to terms. I use best effort to extract the proper meaning. I quote the arguments as they originally appeared inside the paper. However, I rearrange them a bit in order to put related things closer together. I retain the original numbering so that you can compare to the original paper. I also tried to be similar brief with my arguments. Now proof-reading the post, I see that I failed with that. Sorry, but that's as brief as I can provide serious counterargument. I broadly try to classify arguments in various levels of "True" vs "Wrong", so you may take this as an ultra-short reply.

So let's start with Arguments related to the log storage system. In general, the paper is right that there is no real log storage system (like, for example, the Windows Event Log). Keeping logs only in sequential text files definitely has disadvantages. Syslog implementations like rsyslog or syslog-ng have somewhat addressed this by providing the ability to use databases as storage backends (the commercial syslog-ng fork also has a proprietary log store). This has some drawbacks as well. The paper proposes a new proprietary indexed syslog message store. I kind of like this idea, have even considered to write something like this as an optional component for rsyslog (but had no time yet to actually work on it). I am not convinced, though, that all systems necessarily need such a syslog storage subsystem.

With that said, now let's look at the individual arguments:

5. Reading log files is simple but very inefficient. Many key log operations have a complexity of O(n). Indexing is generally not available.

True. It just needs to be said that many tools inside the tool chain only need sequential access. But those that need random access have to pay a big price. Please note, however, that it is often only necessary to "tail" log files, that is act on the latest log entries. This can be done rather quickly even with text files. I know both the problems and the capabilities, because Adiscon LogAnalyzer, in which I am involved, is a web-based analysis and reporting tool capable of working on log files. Paging is simple, but searching is slow with large files (we recommend databases if that is often required). Now that I write that, a funny fact is that one of the more important reasons for creating rsyslog was that we were unhappy with flat text files (see rsyslog history doc). And so I created a syslogd capable of writing to databases. Things seem to be a bit cyclic, though with a different spin ;)

8. Access control is non-existent. Unless manually scripted by the administrator a user either gets full access to the log files, or no access at all.

Mostly True and hard to make any argument against this (except, of course, if you consider database back ends as log stores, but that's not the typical case).

10. Automatic rotation of log files is available, but less than ideal in most implementations: instead of watching disk usage continuously to enforce disk usage limits rotation is only attempted in fixed time intervals, thus leaving the door open to many DoS attacks.

Partly True, at least in regard to current practice. Rsyslog, for example, can limit file sizes as they are written ("outchannel action"), but this feature is seldomly used and due to be replaced by a better one. The better one is partly implemented but received no priority because nobody in the community flagged this as an urgent requirement. As a side-note: Envision that journald intends to shrink the log and/or place stricter restrictions on rate-limiting when disk space begins to run low. If I were an attacker, I would simply begin to fill the disk then, and make journald swipe out the log store for me.

11. Rate limiting is available in some implementations, however, generally does not take the disk usage or service assignment into account, which is highly advisable.

It needs to be said what "rate limiting" means. I guess it means preventing an application from spamming the logs with frequently repeated messages. This feature is available  in rsyslog. It is right that disk usage is not taken into account (see comment above on implications). I don't know what "service assignment" means in this context, so I don't comment on that one. Rate limiting is more than run-away or spamming processes. It is a very complex issue. Rsyslog has output rate limiting as well, and much more is thinkable. But correct, current rate limiting looks at a number of factors but not the disk assignment. On the other hand, does that make sense, if e.g. a message is not even destined to go to the disk?

12. Compression in the log structure on disk is generally available but usually only as effect of rotation and has a negative effect on the already bad complexity behaviour of many key log operations.

Partly True. Rsyslog supports writing in zip format for at least one and a half year (I am too lazy to check the ChangeLog). This provides huge savings for those that turn on the feature. Without doubt, logs compressed in this way are much harder to process in real-time.
 

7. Log files are easily manipulable by attackers, providing easy ways to hide attack information from the administrator

Misleadingly True. If thinking of a local machine, only, this is true. However, all security best practices tell that it is far from a good idea to save logs on a machine that is publicly accessible. This is the reason that log messages are usually immediately sent do some back end system. It is right that this can not happen in some setup, especially very small ones.

My conclusion on the log store: there definitely is room for improvement. But why not improve it within the existing frameworks? Among others, this would have the advantage that existing methods could be used to decide what needs to be stored inside the log store. Usually, log contain noise events that administrators do not want to log at all, because of the overhead associated with them. The exists best practices for the existing tool chain on how to handle that.

Now on to the other detail topics:

1. The message data is generally not authenticated, every local process can claim to be Apache under PID 4711, and syslog will believe that and store it on disk.

9. The meta data stored for log entries is limited, and lacking key bits of information, such as service name, audit session or monotonic timestamps.

Mostly wrong. IMHO, both make up a single argument. At the suggestion of Lennart Poettering, rsyslog can force the pid inside the TAG to match the pid of the log message emitter - for quite a while now. It is also easy to add additional "trusted properties". I made an experimental implementation in rsyslog yesterday. It took a couple of hours and the code is available as part of rsyslog 5.9.4. As a side-note, the level of "trust" one wants to have in such properties needs to be defined - for truly trusted trusted properties some serious cryptography is needed (this is not specified in the journald proposal nor currently implemented in rsyslog).

2. The data logged is very free-form. Automated log-analyzers need to parse human language strings to a) identify message types, and b) parse parameters from them. This results in regex horrors, and a steady need to play catch-up with upstream developers who might tweak the human language log strings in new versions of their software. Effectively, in a away, in order not to break user-applied regular expressions all log messages become ABI of the software generating them, which is usually not intended by the developer.

Trivial (I can't commit myself to a "True" or "Wrong" on such a trivial finding). Finally, the authors have managed to describe the log analysis problem as we currently face it. This is not at all a syslog problem, it is problem of development discipline. For one, syslog has "solved" this issue with RFC5424 structured data. Use it and be happy (but, granted, the syslog() API currently is a bit problematic). The real problem is the missing discipline. Take, for example, the Windows Event Log. The journald proposal borrows heavily on its concepts. In Windows Event Log, there is a developer-assigned unique ID within the application's reserved namespace available. The combination of both app namespace (also automatically created) and ID together does exactly the same thing as the proposed UUID. In Windows Event Log, there are also "structured fields" available, but in the form of an array (this is a bit different from name-value pairs but far from totally different). This system has been in place since the earliest versions of Windows NT, more than 15 years ago. So it would be a decent assumption that the problem described as a syslog problem does not exist in the Windows world, right (especially given the fact that Windows purposefully does not support syslog)? Well, have a look at the problems related to Windows log analysis: these are exactly the same! I could also offer a myriad of other samples, like WELF, Apache Log Format, ... The bottom line is that developer discipline is not easy to achieve. And, among others, a taxonomy is actually needed to extract semantic meaning from the logged event. It probably is educating to read the FAQ for CEE, a standard currently in development that tries to somewhat solve the logging mess (wait a moment: before saying that CEE is a bunch of clueless morons, please have a look at the CEE Board Members first).

3. The timestamps generally do not carry timezone information, even though some newer specifications define support for it.

Partly Wrong. High-Precision timestamps are available for many years and default in rsyslog. Unfortunately, many distros have turned them off, because they break existing tools.  So in current practice this is a problem, but it could be solved by deleting one line in rsyslog.conf. And remember that if that causes trouble to some "vital" tool, journald will break that tool even more. Note that some distros, like Gentoo, already have enabled high precision timestamps.
 

4. Syslog is only one of many log systems on local machines. Separate logs are kept for utmp/wtmp, lastlog, audit, kernel logs, firmware logs, and a multitude of application-specific log formats. This is not only unnecessarily complex, but also hides the relation between the log entries in the various subsystems.

Rhetorically True - but what why is that the failure of syslog? In fact, this problem would not exist if developers had consistently used syslog. So the problem is not rooted in syslog but rather in the fact that syslog is not being used. Lesson learned: even if standards exist, many developers simply ignore them (this is also an interesting argument in regard to problem number #2, think about it...).
 

13. Classic Syslog traditionally is not useful to handle early boot or late shutdown logging, even though recent improvements (for example in systemd) made this work.

True - including that fact that systemd already solved that problem.
 

14. Binary data cannot be logged, which in some cases is essential (Examples: ATA SMART blobs or SCSI sense data, firmware dumps).

Wrong, the short answer is: it can be logged, but must be properly encoded. In the IETF syslog working group we even increased the max message sizes for this reason (actually, there is no hard limit anymore).

The longer, and more correct, answer is that this is a long-standing discussion inside the logging world. Using that view, it is hard to say if the claim is true or false; it often even is argued like being a religion. Fact is that the current logging toolset does not work well for binary data (even encoded). This is even the case for the Windows Event Log, which supports binary data. In my view, I think most logging experts lean towards the side that binary data should be avoided and, if unavoidable, must be encoded in a text-friendly way. A core problem with the usefulness of binary data is that it often is hard to decode, and even more to understand, on the non-native platform (remember that the system used during analysis is often not the system where the event was initially recorded).
 

6. The syslog network protocol is very simple, but also very limited. Since it generally supports only a push transfer model, and does not employ store-and-forward, problems such as Thundering Herd or packet loss severely hamper its use.

Wrong,  missing all improvement made in the past ten years. There is a new RFC series which supports TLS-secured reliable transmission of syslog messages and which permits to place fine-grain access control on who can talk with whom inside a relay chain. UDP syslog is still available and is so for good reason. I cannot dig into the details here, part of that reasoning is on the same grounds why we use audio more often over UDP than TCP. Using UDP syslog is strictly optional and there are few scenarios where it is actually needed. And, if so, the "problem" mentioned is actually a "solution" to a much more serious problem not even mentioned in the journald paper. For a glimpse at these problems, have a lock at my blog post on the "reliability problem". Also, store-and-foward is generally available in rsyslog via action queues, failover handling and a lot of other things. I admit that setting up a complex logging system, sometimes requires an expert. On the "loss issue", one may claim that I myself say that plain TCP syslog is not totally lossless. That claim is right, but the potential loss Window is relatively small. Also, one can use different protocols that solve the issue. In rsyslog, I introduced proprietary RELP for that very reason. There is also completely lossless RFC3195, which is a great protocol (but without future). It is supported by rsyslog, but only extremely few other projects implement it. The IETF (including me) assumes that RFC3195 is a failure - not from technical fact but in the sense that it was too far away from the usual logging practice to be picked up by enough folks. [Just to avoid mis-intepretation: contrary to RFC3195, RELP is well alive, well-accepted and in widespread use. It is only RFC3195 that is a failure.]

Concluding my remarks, I do not see anything so broken in syslog that it can only be fixed by a total replacement of technology. Right contrary, there is a rich tool set and expertise available. Existing solutions, especially in the open source world, are quite modular and can easily be extended. It is not even necessary to extend existing projects. A new log store, for example, could also be implemented by a new tool that imports a decent log format from stdin to a back end data store. That would be easily usable not only from rsyslog but from any other tool that is part of the current log tool chain. For example, it may immediately consume Apache or other application logs (of course, such a tool would require proper cryptography to be used for cryptographic tasks...). There is also need for a new logging API - the catch-all syslog() call is clearly insufficient (an interesting detail fact is that journald promises to retain syslog() as a first-class logging interface -- that means journald can solve none of the issues associated with that API, especially in regard to claim #2).

So extending existing applications, or writing new ones that tightly integrate into the existing toolset is the right thing to do. One can view journald as such an extension. However, this extension is somewhat problematic as its design document tells that it intends to replace the whole logging system. Especially disturbing is that the reasoning, as outlined above, essentially boils down to a new log store and various well-known mostly political problems (with development discipline for structured formats right at the top of them). Finally, the proposal claims to provide more security, but fails to achieve at least the level that RFC5848 syslog is able to provide. Granted, rsyslog, for example, does not (yet) implement RFC5848. But why intends journald to implement some home-grown pseudo security system when a standard-based method designed by real crypto experts is available? I guess the same question can be applied to the reasoning for the journald project at large.

Let me conclude this posting with the same quote I started with:
 

Syslog has been around for ~30 years, due to its simplicity and ubiquitousness it is an invaluable tool for administrators. However, the number of limitations are substantial, and over time they have started to be serious problems:

Mostly Wrong. But it is true that syslog is an invaluable tool,especially in heterogeneous environments.

Links to additional information

• Remote Syslog
• Syslog Messages Classification
• Pipes in syslog
• syslog spoofing
• Syslog Configuration Examples
• Syslog Configuration Debugging
• Syslog Internals

NEWS CONTENTS

• 20120420 : RFC5424 or cee-enhanced syslog? ( April 20, 2012 , Rainer's Blog )
• 20111125 : Linux syslog may be on way out by Brian Proffitt ( November 22, 2011 , ITworld )
• 20080605 : Red Hat 5.1 now uses rsyslog ( Jun 5, 2008 )
• 20071202 : Perl newsyslog by Aaron Scarisbrick ( freshmeat.net )
• 20070628 : rsyslog 1.14.0 (Development) by Rainer Gerhards ( rsyslog 1.14.0 (Development), Jun 28, 2007 )
• 20070105 : An Overview of the syslog Protocol Overview of Syslog by Anand Deveriya. ( An Overview of the syslog Protocol Overview of Syslog, Jan 5, 2007 )
• 20070105 : Understanding system log files on a Solaris ( Understanding system log files on a Solaris, )
• 20070105 : Newlogcheck ( )
• 20030325 : Introduction to Syslog Protocol ( MonitorWare )
• 20030325 : 1.1 Calculating Priority Value ( )
• 20030325 : References ( )
• 20020717 : Sampling syslog software ( Sampling syslog software, Jul 17, 2002 )
• 210219 : Should I use rsyslogs new or old config style ( Should I use rsyslog's new or old config style, )

Old News ;-)

[April 20, 2012] RFC5424 or cee-enhanced syslog?

April 20, 2012 | Rainer's Blog

I recently got a question if it would be better to implement RFC5424 or cee-enhanced syslog in a new project. This sounds like a though question, but looking at the details helps to ease it a bit.

First of all, RFC5424 has a far broader scope than cee-enhanced syslog. RFC5424 is the base RFC of a new series of syslog RFCs that solve many problems, among them (this list for sure will not be complete):

•finally a standardized syslog format (no standard exists before, just a weak understanding)
•supports multiple transports, e.g. UDP and TLS (all in their own RFCs)
•solves some long-standing nits, like the date format (RFC5424 supports high precision and demands a year as part of the timestamp)
•increases the syslog message size and permits operators to set an (almost) arbitrary upper limit for message sizes
•supports structured logging via so-called "structured data"
In contrast, cee-enhanced syslog only cares about the structured logging feature - none of the others is addressed with it. This is the case because its intent is different: the goal here is to have a format that can work with existing technology and still provide the best possible structured logging solution. Depending on circumstances, "best possible" can be quite different. For example, in a scenario where it needs to work with crufty old syslogds, it can mean that message content can not be longer than 1k, a serious restriction. Even more, it is envisioned that cee-enhanced syslog will also work over non-syslog transport, and may even be transported by simply copying (and merging) files.

CEE has long thought about using RFC5424 structured data format for its structured logging needs. However, that would have limited its ability to work with all these older (and deployed) syslog implementations. Thus, the decision was made to use text that purely resides inside the message part and, without transformation, can be used in legacy systems (with given restrictions, of course).

So now back to the question: what to implement? With this facts, I think it boils down to "both". To use the benefits of modern-day syslog (as described above), you definitely need to implement RFC5424 and its helper RFCs (like 5425 for TLS). HOWEVER, the question remains if one should use structured data or cee-enhanced syslog. This, I have to admit, is indeed a though question.

My personal opinion, which can be totally wrong and is solely based on my experience (aka "gut feeling" ;)), cee-enhanced syslog sounds more promising for structured data. The main reason for this is that the format has received very good responses from major players and a lot of work is currently being done by a lot of folks driving important projects in the logging environment. There is also a lot of supporting libraries upcoming AND some of them already available. Most importantly, this effort is thightly integrated with Mitre and it probably is not to far-fetched to assume that cee-enhanced syslog will appear on some purchasing checklists in the not so distant future. RFC5424 structured data does not have this broad backing. So while cee-enhanced syslog is a very fresh project, my personal assumption is that it will take off.

But again, that does not mean that RFC5424 shall not be implemented. I think RFC5424 technology is vital for a number of reasons. Of course, things like TLS can also be obtained without RFC5424, but not in a standards-compliant way. So to ensure interoperability, going the 5424 way is a good idea.

[Nov 25, 2011] Linux syslog may be on way out By Brian Proffitt

It is naive to link binary format with better security. Instantly the hacking tools will be created to "sanitize" binary logs in sophisticated way. So instead of improvement of security we will get just the next level of intruders vs. defenders war. Bob Simpson in his comment below made an important point" the author of the proposal think locally rather the globally. And the key about logs security is thinking globally. But still it is true that the current syslog implementation which is coming from Sendmail is outdated and can be improved. But is can be improved without abandoning text format and throwing the child with the dirty bathwater. Remote syslog is the only real security enhancing mechanism for syslog but "chain-signing" records with some certificate can also be useful on local level and complicates tampering. The efforts spend on syslog probably be extended to shell history. Shell history also can benefit from some "anti-tampering" mechanisms. One easy thing is to make it "unshrinkable" via special filesystem attributes mechanisms.

November 22, 2011 | ITworld

In an effort to foil crackers attempts to cover their tracks by altering text-based syslogs, as well as improve the syslog process as a whole, two Red Hat developers are proposing a new binary-based tool called The Journal that could replace the syslog daemon in as early as the Fedora 17 release.

And believe you me, some people are less than enthused by the proposed solution.

Developers Lennart Poettering and Kay Sievers are proposing that the current 30-year-old syslog system is inefficient and too easy to misread and hack to properly perform even its most basic function: store a log of system events on a given Linux box.

This is largely due to the free-form nature of the syslog, which basically accepts text strings in whatever form the application or daemon on the Linux system chooses to send. So, one daemon may send information about an event in one way, and another daemon in a completely different way, leaving it up to the human reader to parse the information in a useful manner. Automated log analyzer tools can help with this, but in a detailed description of The Journal, Poettering and Sievers wrote:

"The data logged is very free-form. Automated log-analyzers need to parse human language strings to a) identify message types, and b) parse parameters from them. This results in regex horrors, and a steady need to play catch-up with upstream developers who might tweak the human language log strings in new versions of their software. Effectively, in a away, in order not to break user-applied regular expressions all log messages become ABI of the software generating them, which is usually not intended by the developer."

That's just one of 14 points the two developers have highlighted as problems with the current syslog system. Others include:

• Syslog data is not authenticated.
• Syslog is only one of many logging systems on a Linux machine.
• Access control to the syslogs is non-existent.
• Disk usage limits are only applied at fixed intervals, leaving systems vulnerable to DDoS attacks.

And so on. Poettering and Sievers highlighted one very topical problem with the syslog system to drive their points about a needed change to syslog home:

"For example, the recent, much discussed kernel.org intrusion involved log file manipulation which was only detected by chance."

With these factors in mind, Sievers and Poettering have come up with The Journal daemon, which will store data from system events in binary--not text--form as a list of key-value pairs that includes hashing for additional security.

This is not the first time these two developers have proposed such sweeping changes to the Linux system infrastructure. Poettering is the developer who invented the systemd daemon that replaced the System V init daemon on Linux, as well as invented the PulseAudio sound server. Sievers was most recently one of the Fedora Project team members who proposed to move all executable files into the /usr/bin directory and their libraries into /usr/lib or /usr/lib64, as needed.

With this binary implementation, The Journal daemon can enable the addition of metadata to each system event, such as the process ID and name of the sender, user and group IDs, and other key system data.

"Inspired by udev events, journal entries resemble environment blocks. A number of key/value fields, separated by line breaks, with uppercase variable names. In comparison to udev device events and real environment blocks there's one major difference: while the focus is definitely on ASCII formatted strings, binary blobs as values are also supported--something which may be used to attach binary meta data such as ATA SMART health data, SCSI sense data, coredumps or firmware dumps. The code generating a journal entry can attach as many fields to an entry as he likes, which can be well-known ones, or service/subsystem/driver specific ones."

If all of this seems a bit familiar to developers, see if this rings a bell: a lot of the effort here by Poettering and Sievers was inspired by the key/value, hash, and metadata provided to developers who use the git version control system.

Not only will implementing The Journal make a Linux system more secure (as unauthorized log entries or unexpected data field entries will immediately be flagged by the journal daemon), its inventors hope to actually reduce the footprint of the logging system on Linux by unifying all log systems on a Linux machine and efficiently restructuring the data.

"It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that."

But not everyone is thrilled with the proposal. Poettering and Sievers anticipated that many developers and system admins would be unhappy with The Journal's use of UUIDs to identify messages--as evidenced by their tongue-in-cheek attention to the issue in the FAQ section of their proposal document.

But many of the objections voiced on Linux Weekly News, where the proposal was first highlighted, lament the replacement of a simple text-based system with a binary data format that will rely on one tool--The Journal--which in turn will only be available with the systemd daemon.

Several commenters picked up on this entry in The Journal proposal FAQ:

"Will the journal file format be standardized? Where can I find an explanation of the on-disk data structures?

"At this point we have no intention to standardize the format and we take the liberty to alter it as we see fit. We might document the on-disk format eventually, but at this point we don't want any other software to read, write or manipulate our journal files directly. The access is granted by a shared library and a command line tool. (But then again, it's Free Software, so you can always read the source code!)"

That entry, more than any other in the proposal document, generated a lot of controversy, as many LWN commenters objected to the idea of using a non-standard format for The Journal's data. Backwards compatibility was also a big point of concern.

"It's a shame that we will lose the simplicity of the plain-text syslog format. But syslogs are usually compressed using gzip anyway. So essentially for me, all this means is that I use <magic-lennart-tool> instead of gzcat as the first part of my shell command, wrote commenter C. McCabe. "The big issue that I see is that a lot of system administrators will treat this as magic security dust, and not realize that they need to periodically save those hashes to a remote (and secure!) system in order to get any security benefit.

"I also hope Lennart and co. realize the absolute necessity of backwards compatibility for the on-disk format," McCabe added. "It would really embitter a lot of system administrators if their old logs became unreadable after upgrading to the shiniest new version. But assuming this is managed well, I don't see any reason why this couldn't be a good idea."

How this plays out in the broader Linux community will be interesting, to be sure. I personally find it notable that Fedora (and its commercial parent Red Hat) now seems to be the project where many internal infrastructure changes to the Linux operating system are getting implemented, even as distros like Ubuntu focus on the interface and user space.

This is not a judgmental statement, but rather an observation. Linux is clearly undergoing some significant evolutionary changes and shedding some of its UNIX legacy. What remains to be seen is how these changes will affect Linux as it moves ahead.

Selected comments

alevesely_tw273908198

I'm not sure why the design proposed would be definitely better than upgrading syslog() so as to cover recent standardizations such as

* RFC 5424: The Syslog Protocol * RFC 5674: Alarms in Syslog * RFC 5675: Mapping Simple Network Management Protocol (SNMP) Notifications to SYSLOG Messages * RFC 5676: Definitions of Managed Objects for Mapping SYSLOG Messages to Simple Network Management Protocol (SNMP) Notifications * RFC 5848: Signed Syslog Messages

Proposer of an enhancement should take the trouble to standardize it, beside coding, experimenting, and evangelizing it.

BobSimpson

Is this a joke? Or is it someone just trying to push their ideology of what they think should be done to the rest of the world to make their idea a standard?

Doing something like this would be a sure way for Linux to shoot itself in the foot. For evidence, one only needs to look as far as Microsoft who insists on doing it their special way and expecting everyone else to do what they deem as "good". The concept of syslog messages are that they are meant to be 'open' so disparate systems can read the data. How to you propose to integrate with large syslog reporting/analysis tools like LogZilla (http://www.logzilla.pro)?

The authors are correct that a format needs to be written so that parsing is easier. But how is their solution any "easier"? Instead, there is a much more effective solution available known as CEE (http://cee.mitre.org/) that proposes to include fields in the text.

> Syslog data is not authenticated. If you need that, then use TLS/certificates. when logging to a centralized host.

>Syslog is only one of many logging systems on a Linux machine. Surely you're aware of syslog-ng and rsyslog.

Access control to the syslogs is non-existent.

> To locally stored logs? Maybe (if you don't chown them to root?)

> But, if you are using syslog-ng or rsyslog and sending to a centralized host., then what is "local" to the system becomes irrelevant.

Disk usage limits are only applied at fixed intervals, leaving systems vulnerable to DDoS attacks. > Again, a moot point if admins are doing it correctly by centralizing with tools like syslog-ng, rsyslog and LogZilla.

>"For example, the recent, much discussed kernel.org intrusion involved log file manipulation which was only detected by chance." Oh, you mean they weren't managing their syslog properly so they got screwed and blamed their lack of management on the protocol itself. Ok, yeah, that makes sense.

They also noted in their paper that " In a later version we plan to extend the journal minimally to support live remote logging, in both PUSH and PULL modes always using a local journal as buffer for a store-and-forward logic" I can't understand how this would be an afterthought. They are clearly thinking "locally" rather than globally. Plus, if it is to eventually be able to send, what format will it use? Text? Ok, now they are back to their original complaint.

All of this really just makes me cringe. If RH/Fedora do this, there is no way for people that manage large system infrastructures to include those systems in their management. I am responsible for managing over 8,000 Cisco devices on top of several hundred linux systems. Am I supposed to log on to each linux server to get log information?

BlackSteven_YahHVZXTW

1. It's RedHat so if they actually go with it it will be forced to the vast majority of folks that work with Linux in a corporate environment.

2. It's RedHat so it doesn't need to be grep-able. They cater to current and former Windows users who call themselves "geeks" because they use an Operating System that is harder to use. (Linux doesn't have to be hard to use -- but don't tell that to RedHat.)

3. The guys behind this have been behind a number of really hideous ideas. One of them actually proposed dropping /usr. (Functionally dropping /usr by dropping /bin /sbin and /lib.) If I can't unmount /usr, then it doesn't _functionally_ exist.

4. It'll only be available with the "systemd" startup mechanism. This makes it totally pointless, as there's no way Debian derived distributions will be forcing folks to use that. It'll just be another piece of RedHat goofiness. Since prior RedHat goofiness kept me away from all RedHat derived distributions, this has little impact for me personally.

5. Many of the issues he has problems with are either generally useful -- human-readable logs readable with any tool -- or mostly an issue when not using syslog-ng and a log vault. Do you think your system was compromised? What does your log vault say?

6. If this system doesn't have a log vault facility it is only a matter of time before it is circumvented when root is compromised. When root is compromised anything touchable on the system is suspect. Their use of a shifting non-standard data format does nothing to make the data safer and it breaks in-place upgrades. What it means is that someone else will make a shared library that can read/write all versions of their logs and crackers will use that shared library. The fact that it is open-source and written in C means that having one library which links to multiple implementations of their "The Journal" is trivial. (Hello, C preprocessor!)

7. Remember that -- since this is RedHat -- they have a long history of not recommending in-place upgrades. If this breaks in-place upgrades or makes long-term archival of logs impossible there is no reason to think that it will stop them.

8. Portable software will need to support both the new system and continue to support SysLog. There's no way the BSDs would be migrating to this, even if the existing viable commercial Unix world decides to promptly oblige the whims of RedHat. More than that, basic syslog functionality can be easily faked on non-Unix-like environments without a syslog daemon and without breaking the log format. This means -- when using syslog -- the documentation for the product needs to mention the alternate location of the log but the actual documentation for log data is the same. This is not so with this new tool, where there's a different data format and different way to access the data.

Is it a good idea? Absolutely not. Will it stop them? They're RedHat -- they're too big to fail.

Here's another idea. What's the difference between forcing this additional data in to another tool which we will end up using through a text-based interface (so we can grep it) and actually proposing a standard for how to send log data to syslog which will support enhanced storage and retrieval on the backend? You could even offer a convenient C function to take the arguments supported by "The Journal" and pass the data in the correct format to syslog.

Brendan Edmonds

One of the big problems with implementing a binary base log file system is as mentioned already, only one tool can read it correctly. And you lose the security if someone finds out how the data is stored. It's the exact same thing "Problem" as a text based log file. The other problem, a lot has to change in order for programs to work with the new system, i'e dmesg needs to be changed.

Also I can not see how you can use the same methods as before to clean the logs, and to backup these logs. For example what happened if you wanted to view that log file in Windows, how would you do this. From a tool point of view you need the correct security tokens, etc.

You also have to remember routers use the syslogd protocol to send messages to a unix system. How will this be handled?

I don't like the move, it defeats the whole point of UNIX. Every thing in UNIX is a text file. Anyone can add to it and anyone can remove lines from it. It is up to the kernel to control who can do what and the program, who can do what. The point about the file being text only is mute as normally only root and write to the file and a binary file has the same "problem".

JoeGiles:

"I don't like the move, it defeats the whole point of UNIX. Every thing in UNIX is a text file"

O'Rly?

When was the last time you took a look at wtmp or btmp or saXX files with vi? Not every file in Linux is a text file!

I think its a great idea as long as its executed properly!

NickG:

Personally, I love all the changes coming into Fedora. The filesystem is archaic, log files outright useless in their current implementation (kernel.org hack proves this), and systemd is chock full of advanced functionality. My only gripe is with the and the possibility of some logs becoming no longer backwards-compatible.

Other than that, if PulseAudio and systemd are any indication, we'll hopefully be seeing these changes filter into the other distros as well soon.

IFMAP_ORG_tw389617356:

A better approach would be to use an identity-centric open standard such as IF-MAP which is already used by many security vendors to integrate security information from multi-vendor environments.

See www.if-map.org

[Jun 5, 2008] Red Hat 5.1 now uses rsyslog

From release notes for Red Hat Enterprise Linux 5.2

rsyslog is an enhanced multi-threaded syslogd daemon that supports the following (among others):

• MySQL
• syslog/tcp
• RFC 3195
• permitted sender lists
• filtering on any message part
• more granular output format control

rsyslog ...ith the stock syslogd, and can be used as a replacement in most cases. Its advanced features make it suitable for enterprise-class, encrypted syslog relay chains; at the same time, its user-friendly interface is designed to make setup easy for novice users.

For more information about rsyslog, refer to http://www.rsyslog.com/.

[Dec 2, 2007] Perl newsyslog by Aaron Scarisbrick

freshmeat.net

newsyslog is a faithful Perl rewrite of the MIT newsyslog utility, with a number of features taken from the FreeBSD and NetBSD variants of newsyslog. It archives log files based on size, date or interval, and can optionally compress archives with gzip or bzip2. Complete documentation is available via "perldoc newsyslog.pl".

[Jun 28, 2007] rsyslog 1.14.0 (Development) by Rainer Gerhards

About: Rsyslog is an enhanced multi-threaded syslogd. Among other features, it offers support for reliable syslog over TCP and RFC 3195, writing to MySQL databases, fully configurable output formats (including great timestamps), the ability to filter on any part of the syslog message, and on-the-wire message compression. It is designed as a drop-in replacement for stock syslogd and thus is able to work with the same configuration file syntax. Of course, some enhanced features require changing the configuration file, but in general, this should be fairly easy.

Changes: Support for IPv6 has been added and some minor cleanups plus a fix for the Red Hat init script were applied. Currently, IPv6 is implemented for UDP only, TCP will follow shortly. IPv6 support should be considered experimental. It is not recommended that this release be used in production.

[Jan 5, 2007] An Overview of the syslog Protocol Overview of Syslog by Anand Deveriya.

This chapter presents an overview of the syslog protocol and shows you how to deploy an end-to-end syslog system. You'll learn about the syslog architecture as well as the issues in deploying syslog servers in Linux and Windows OSs with a focus on their relevance in a Cisco environment.

Understanding system log files on a Solaris

Solas use the /var directory to store logs and other local files so that the operating system can support other directories being mounted as read only, sometimes from file servers elsewhere on the network. The /var directory is thus often on a partition that is local to the system.

All of the log files described below can be found in subdirectories under /var. There may be other application-specific log files that you will also need to inspect. However, it is beyond the scope of this implementation to describe all of the log files that you might want to inspect for your specific Solaris installation.

Because log files often provide the only indication of an intrusion, intruders often attempt to erase any evidence of their activities by removing or modifying the log files. For this reason, it is very important that your log files be adequately protected to make it as difficult as possible for intruders to change or remove then. See the practice "Managing logging and other data collection mechanisms" for more information on this topic.

Newlogcheck

My central loghost machine uses a modified version of logcheck.sh that I wrote named (imaginatively) newlogcheck.sh. The modified version calls another script I wrote that sorts the output of the "logtail" by individual hosts into separate portions of the final report. The perl script attempts to avoid duplication of log messages by printing each log message only once, reporting how many times the event was reported.

This approach dramatically reduces the size of your logcheck reports, and sorting it by host makes it easy to read.

Check out a sample newlogcheck report

You need to configure your logcheck settings yourself, read the README that comes with logcheck from psionic.com, then the one included with my scripts. Copy my newlogcheck.sh and sort_logs.pl into your logcheck dir, and run newlogcheck.sh instead of logcheck.sh for reports.

If you're ready to go ahead, get the tarred/gzipped file here.

[Mar 25, 2003] Introduction to Syslog Protocol

MonitorWare

This paper is not an in depth paper about syslog. It simply gives you an overview and a broader picture about the Syslog Protocol and its architecture. If you are interested in in-depth details about Syslog, I would strongly suggest you to go through RFC: 3164.

What is Syslog?

Syslog is a protocol that allows a machine to send event notification messages across IP networks to event message collectors - also known as Syslog Servers or Syslog Daemons. In other words, a machine or a device can be configured in such a way that it generates a Syslog Message and forwards it to a specific Syslog Daemon (Server).

Syslog messages are based on the User Datagram Protocol (UDP) type of Internet Protocol (IP) communications. Syslog messages are received on UDP port 514. Syslog message text is generally no more than 1024 bytes in length. Since the UDP type of communication is connectionless, the sending or receiving host has no knowledge receipt for retransmission. If a UDP packet gets lost due to congestion on the network or due to resource unavailability, it will simply get lost - no one would know about it!!

What is Syslog Daemon?

A Syslog Daemon or Server is an entity that would listen to the Syslog messages that are sent to it. You cannot configure a Syslog Daemon to ask a specific device to send it Syslog Messages. If a specific device has no ability to generate Syslog Messages, then a Syslog Daemon cannot do anything about it. To make this thing clear, you can consider a Syslog Server or Syslog Daemon as a TV which can only display you the program that is currently running on a specific channel. You cannot ask another station to send a new program on that channel.

Some Background

Syslog Protocol was created for use by Unix Operating Systems. Using Syslog, a remote Unix host could, in effect, keep track of the general well being of any other Unix host. Any application can generate Syslog Compliant messages to send the information over the network. Since each process, application and operating system was written somewhat independently, there is little uniformity to the content of syslog messages. For this reason, no assumption is made upon the formatting or contents of the messages. The protocol is simply designed to transport these event messages. One of the fundamental design considerations of the syslog protocol was its simplicity. No stringent coordination is required between the transmitters and the receivers. Indeed, the transmission of syslog messages may be started on a device without a receiver being configured, or even actually physically present. Conversely, many devices will most likely be able to receive messages without explicit configuration or definitions. This simplicity has greatly aided the acceptance and deployment of syslog [1]

Format of a Syslog Packet

The full format of a Syslog message seen on the wire has three ditinct parts.

1. PRI

2. HEADER

3.MSG.

The total length of the packet cannot exceed 1,024 bytes, and there is no minimum length

1. PRI

The Priority part is a number that is enclosed in angle brackets. This represents both the Facility and Severity of the message. This number is an eight bit number. The first 3 least significant bits represent the Severity of the message (with 3 bits you can represent 8 different Severities) and the other 5 bits represent the Facility of the message. You can use the Facility and the Severity values to apply certain filters on the events in the Syslog Daemon. Note that Syslog Daemon cannot generate thse Priority and Facility values. They are generated by the applications on which the event is generated. Following are the codes for Severity and Facility. Please note that the codes written below are the recommended codes that the applicatoins should generate in the specified situations. You cannot, however, be 100 % sure if it really is the correct code sent by the application. For example: An application can generate a numerical code for severity as 0 (Emergency) when it should have generated 4 (Warning) instead. Syslog Daemon can not do anything about it!! It will simply receive the message as it is.

a) Severity Codes

The Severity code is the severity of the message that has been generated. Following are the codes for Severity:

Numerical Code	Severity
0	Emergency: system is unusable
1	Alert: action must be taken immediately
2	Critical: critical conditions
3	Error: error conditions
4	Warning: warning conditions
5	Notice: normal but significant condition
6	Informational: informational messages
7	Debug: debug-level messages

b) Facility Codes

The facility is the application or operating system component that generates a log message.Following are the codes for facility:

Numerical Code	Facility
0	kernel messages
1	user-level messages
2	mail system
3	system daemons
4	security/authorization messages
5	messages generated internally by syslogd
6	line printer subsystem
7	network news subsystem
8	UUCP subsystem
9	clock daemon
10	security/authorization messages
11	FTP daemon
12	NTP subsystem
13	log audit
14	log alert
15	clock daemon
16	local use 0
17	local use 1
18	local use 2
19	local use 3
20	local use 4
21	local use 5
22	local use 6
23	local use 7

1.1 Calculating Priority Value

The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity. For example, a kernel message (Facility=0) with a Severity of Emergency (Severity=0) would have a Priority value of 0. Also, a "local use 4" message (Facility=20) with a Severity of Notice (Severity=5) would have a Priority value of 165. In the PRI part of a Syslog message, these values would be placed between the angle brackets as <0> and <165> respectively.

2. Header

The HEADER part contains the following things:

a) Timestamp -- The Time stamp is the date and time at which the message was generated. Be warned, that this timestamp is picked up from the system time and if the system time is not correct, you might get a packet with totally incorrect time stamp

b) Hostname or IP address of the device.

3. MSG

The MSG part will fill the remainder of the Syslog packet. This will usually contain some additional information of the process that generated the message, and then the text of the message. The MSG part has two fields:

a) TAG field

b) CONTENT field

The value in the TAG field will be the name of the program or process that generated the message. The CONTENT contains the details of the message.

Some Important Points

1. As mentioned above, since Syslog protocol is UDP based, it is unreliable. It does not guarantee you the delivery of the messages. They may either be dropped through network congestion, or they may be maliciously intercepted and discarded.
2. As mentioned above, since each process, application and operating system was written somewhat independently, there is little uniformity to the content of syslog messages. For this reason, no assumption is made upon the formatting or contents of the messages. The protocol is simply designed to transport these event messages.
3. The receiver of a Syslog packet will not be able to ascertain that the message was indeed sent from the reported sender.
4. One possible problem associated with the above mentioned point is of Authentication. A misconfigured machine may send syslog messages to a Syslog Daemon representing itself as another machine. The administrative staff may become confused because the status of the supposed sender of the messages may not be accurately reflected in the received messages.
5. Another problem associated with point 2 is that an attacker may start sending fake messages indicating a problem on some machine. This may get the attention of the system administrators who will spend their time investigating the alleged problem. During this time, the attacker may be able to compromise a different machine, or a different process on the same machine.
6. The Syslog protocol do not ensure ordered delivery of packets.
7. An attacker may record a set of messages that indicate normal activity of a machine. At a later time, that attacker may remove that machine from the network and replay the syslog messages to the Daemon.

Related Software

The MonitorWare line of products [2] can be used as Syslog Daemons for Windows Operating System to collect Syslog Messages from various devices (including Routers, Fire walls etc). They can also act as relaying servers and can forward the data from one Syslog Daemon to another.

References

[1] http://www.ietf.org/rfc/rfc3164.txt

[Jul 17, 2002] Sampling syslog software

Last week we discussed syslog, a system for handling status messages and logging and we looked briefly at the format of a syslog message. There's a lot more to the standard, and we encourage you to read the relevant IETF standard, RFC 3164, "The BSD syslog Protocol".

The syslog protocol is very useful, but be warned it has its deficiencies: It isn't secure; syslog messages are relatively easy to fake (sending syslog messages greater than the standard maximum of 1,024 bytes has been used in an exploit to hack syslog) and there's no sender validation. Anyway, we will forgo delving any deeper into the bowels of syslog and instead look at syslog products.

Should I use rsyslog's new or old config style

Rainer's Blog

I got a very interesting question on the rsyslog support forums, and I thought I share it, together with the answer, here at a more prominent spot:

After over a decade of using stock bsd syslog, I finally have a need to do some more complicated processing of logs (splitting off Postgres query logs from general Postgres logs), and after looking at other options (basically syslog-ng), I think rsyslog looks like a better fit. I'm mainly in it so I can use regex matching, but thinks like the log queueing and being able to easily move to db storage in the future look good.
Since I'm new, I'd considered that I might get a jump on things by sticking with the newest config syntax. But after doing some googling for examples and looking at the examples in the rsyslog wiki, it seems like maybe the newest syntax might be a bit too new for a beginner - I learn best by example.

Are there any serious downsides to NOT going with the most current syntax?

The answer is that the old syntax is still fully supported by the versions and will probably remain for quite some while (except for some very few exceptions, which we couldn't carry over for good reasons - this is documented in the compatibility docs on the web site). Some parts of it are considered so important that they most probably never will go away. Actually, if you want to do simple things, the old syntax has its advantages. The more complex your processing gets, the more you benefit from the new syntax. But you can mix and match new and old style in almost all cases.

So my suggestion would be to get started using the old syntax and as soon as you begin to do more complex things, you can switch over to the new style. That's actually the way it is designed ;) A good indicator of when it would be benefitial to move to new style is when you begin to use a lot of directives beginning with $, especially if they modify an action. Also, if you move to action queues, I would strongly suggest to use new style. It is far more intuitive an less error-prone.

To provide a bit more background information, there is an important non-technical reason why the classical syntax is remain for a long time: basic syslog.conf format is extremely well known, covered in a lot of text books, taught in numerous courses and used in a myriad of Internet tutorials. So if we would abandon it, we would thrash a lot of people's knowledge and help resources. In short: we would make it much harder for folks that it would actually need to be. This has never been rsyslog philosophy. Providing the ability to changed gradually and with growing needs is a core goal.

Recommended Links

Softpanorama Recommended

Top articles

Sites

• Syslog - Wikipedia, the free encyclopedia
• freshmeat.net Browse project tree - Topic System Logging
• LogAnalysis.Org
  • Device Specific syslog Configurations
• Reliable Delivery for syslog
• Security Issues in Network Event Logging (syslog) Charter
• IETF Syslog Working Group Home Page
• SDSC Syslog Home Page
• PHP- syslog - Manual
• freshmeat.net Project details for syslog-ng
• Syslog-ng faq
• Java Syslog
• Central Loghost Mini-HOWTO
• An Overview of the syslog Protocol Overview of Syslog

  This chapter presents an overview of the syslog protocol and shows you how to deploy an end-to-end syslog system. You'll learn about the syslog architecture as well as the issues in deploying syslog servers in Linux and Windows OSs with a focus on their relevance in a Cisco environment.

RFCs

RFC 3164 (rfc3164) - The BSD Syslog Protocol

rfc3195 Reliable Delivery for syslog

Supporting documents and discussions:

• Transmission of syslog messages over UDP
• Why in-depth Specs
• IETF Syslog Working Group Home Page
• 59 IETF syslog WG

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: February 10, 2021