Syslog Internals

syslog: The UNIX System Logger

syslog is a powerful and easily configurable UNIX system resource. Available since the earliest releases of BSD UNIX, it is now supported on most versions of UNIX. Designed to be the UNIX system logging facility, syslog has always offered not just local logging to files, but also remote logging over the network via standard protocols to mixed vendor systems.

System Logging in UNIX

Logging and error reporting in UNIX have been unnecessarily complex. The following examples illustrate typical approaches for logging messages and reporting errors to the system console.

In a shell script:

% ./mydaemon 2> /dev/console &    # stderr sent to console

In C-code:

fp_console = fopen("/dev/console","w")

if ( ! stat("filename",&stat_buf) )

	/* normal action here */

else  /* error output to console */

      fprintf(fp_console,"Error: cannot open file");

While this may be acceptable for output written directly to your process window, these approaches have potential drawbacks for console I/O:

• the message will be lost on a console-less system
• the message may scroll off the console and be lost
• the operator may not be at the console for immediate action
• a user logged in at the console may receive a message that is confusing or inappropriate
• the message will not be logged for future review
• console output may become garbled by competing processes
• console screen appearance and formatting may be overwritten

In general, system log and error messages should not be written directly to the system console, except under the most urgent of circumstances. For example, a kernel crash and the display of system state information should be directed to the system console.

To remedy non-uniform message reporting and address problems noted above, early releases of the BSD-UNIX operating system introduced the syslog message reporting system. syslog was then ported to major UNIX system implementations as features of BSD-UNIX were adopted. syslog has been implemented in HP-UX since the early releases of the operating system. Work has been done on syslog in the 10.x releases to address performance and standards issues.

Architecture of syslog

The syslog system consists of the following components:

• message format specification: syslog.h

syslog messages are encoded as ASCII strings. Message strings are created throughout the UNIX system. Messages are created at one of a set of possible levels; by setting a threshold one can direct all messages at this or a higher level to given locations. Definitions for syslog messages are in the include file usr/include/syslog.h.

• a set of calls for creating those messages: syslog(3c), logger(1)

Most users create syslog messages through one of the standard interfaces to syslog. The library call syslog(3c) (contained within libc) is a C-code interface to creating message strings. syslog(3c) behaves somewhat like the standard printf() interface. From the command line or within a shell script, one can invoke the command logger(1) to create syslog messages.

• a set of locations from which messages can be read: /dev/log, /dev/klog, UDP port 514

Messages come into syslog from various paths. Every syslog message must be directed to one of a number of communication paths that are read by the syslogd daemon. These communication channels can include:

• a daemon that reads from these locations: syslogd

  The syslogd daemon is a user-space process. syslogd waits for incoming messages and directs the message to possible output locations. A message that is logged (to a file, a logged-on user, or the console) is also formatted and tagged with the system name. The syslogd daemon may also do some filtering of messages, for example to prevent repeated messages from flooding the system.

• a set of locations to which messages can be directed: files, users, etc.

This set of locations may include sockets, files, the system console, and logged-on users. Messages can be directed to one or more of these locations.

• rules for how messages are directed: /etc/syslog.conf

This set of configuration rules determines how messages are logged and the locations to which they are logged. The rules are set by the system administrator in the file /etc/syslog.conf.

Software developers who are creating their own syslog messages will be interested in the path for user-created syslog messages. User messages are typically created through either the syslog(3c) C-code interface or the logger(1) command. These messages are written to the pipe /dev/log, from which they are then read by the syslogd daemon, which directs them to configured output locations.

Message Format

One of the unique characteristics of syslog is that every message is created and logged in the form of a plain text ASCII string. Messages are then directed through pipes and sockets, written to log (ordinary) files, and displayed to users using only standard string handling operations. This presents an easy-to-use and well-understood interface but has a few performance implications that will be discussed later.

Throughout this discussion we will maintain a distinction between what we will refer to as a syslog message and a logged message. A syslog message is the actual ASCII string that is sent to or read from the pipes and sockets that make up the syslog system. It is useful to review this message format to understand some of the capabilities and limitations of syslog. The logged message is the ASCII string that is written to a log file or to a user at a terminal. This is the string the system administrator or an ordinary user will actually see.

The syslog message format is defined in the include file syslog.h. In this file are the definitions for message priority (level and facility) and the flags used to configure the syslog interfaces. Function prototypes for the syslog library calls were added to syslog.h for the HP-UX 10.x releases.

Components of syslog Messages

A syslog message is an ASCII string that consists of :-

a message priority

a timestamp

the message string

Priority is encoded as an ASCII string enclosed by the angle brackets < and > at the beginning of the string. Message priority is the ASCII integer encoding of an 8-bit quantity. This quantity is a combination of a 3-bit field (bits 0 through 2) used for message level and a 5-bit field (bits 3 through 7) used for the message facility. Thus message priority level can have 8 possible values, and message facility up to 32 possible values.

Message levels are defined as an ordered list. If one has set a threshold at a given level, one will receive all messages at this or any higher level. Thus, if you have configured your syslog to log all messages tagged LOG_WARNING, you will also log all messages of higher levels, such as LOG_ERR, LOG_CRIT. The defined message levels, from highest level to lowest, are as follows:

LEVEL	CODE	DESCRIPTION
LOG_EMERG	0	kernel panic
LOG ALERT	1	condition needing immediate attention
LOG_CRIT	2	critical conditions
LOG_ERR	3	errors
LOG_WARNING	4	warning messages
LOG_NOTICE	5	not an error, but may need attention
LOG_INFO	6	informational messages
LOG_DEBUG	7	when debugging a system

The message facility is a tag to identify the originating subsystem of the message. Facility tags are defined in syslog.h; some are reserved for the OS and others are available for users and application developers. The following message facility tags are defined:

FACILITY	CODE	DESCRIPTION
LOG_KERN	(0<<3)	kernel messages
LOG_USER	(1<<3)	random user-level messages
LOG_MAIL	(2<<3)	mail system
LOG_DAEMON	(3<<3)	system daemons
LOG_AUTH	(4<<3)	security/authorization messages
LOG_SYSLOG	(5<<3)	messages generated internally by syslogd
LOG_LPR	(6<<3)	line printer subsystem
LOG_NEWS	(7<<3)	messages generated by the news system
LOG_UUCP	(8<<3)	messages generated by the UUCP system
LOG_CRON	(9<<3)	messages generated by the cron daemon
other codes 10 through 15 are reserved for system use
LOG_LOCAL[0-7]	(16-23<<3)	reserved for local use

Logged Messages

The logged message is the ASCII string that will be written by the syslogd daemon to a user, to the console, or to a log file. The logged message is also a plain text ASCII string, but of somewhat different format. The message priority field is removed; the information in this field is used by syslogd to direct the syslog message according to the configuration rules set in /etc/syslog.conf. The timestamp is the same ASCII-encoded date/time string as before. A new "system" field is next in the string: this is the name of the system (equivalent to uname -n) that has sent the message. This is the local system if the message was locally generated, or a remote system communicating over an Internet domain socket. The message string is the actual text of the message; useful fields such as process ID and a message prefix may be inserted in this message by using the openlog() function.

Creating syslog messages syslog(3c), logger(1)

Most users and software developers will create syslog messages through the standard interfaces syslog(3c) and logger(1). The syslog library functions openlog(), closelog(), and setlogmask() are contained in the libc library; function prototypes are defined in syslog.h. The logger command (/usr/bin/logger) provides similar functionality from the command line or in a shell script.

The syslog call provides an interface with printf()-like string handling to syslog. Here is an example:

syslog(priority,"message_string")

Message priority is a combination of a syslog level and facility tag. The message string has a printf-like interface: one can log ASCII strings using standard % format semantics. In addition, the "%m" format will print the global errno for the originating process. syslog() will return an error if /dev/log cannot be opened. For the 10.x releases syslog() was modified to retry the write to /dev/log if it is busy, up to 20 times with a ¹/₁₀th -second delay.

The function openlog() provides a useful method for controlling the logging process and setting default fields in the message strings. The function is called as follows:

openlog("identifying_string",log_options,default_facility)

where the following parameters can be set:

"identifying_string" is a string that is prepended to each syslog message

log_options can be one or more of the following:

LOG_PID	log process ID with each message (very useful)
LOG_CONS	write to console if unable to send to syslogd
LOG_NDELAY	open connection to syslogd immediately
LOG_NOWAIT	no wait for children logging messages to console

default_facility assigns a facility tag to identify the originating subsystem (very useful)

The following calls are also part of the syslog library interface:

setlogmask(maskpri)--sets the process log priority mask to maskpri and returns the previous priority mask value

closelog()--closes the log file

From the command line or a shell script the command logger(1) can be used. logger parses the command line arguments and calls syslog(3c).

logger [-t tag] [-p priority] [-i] [-f file | message ]

-t tag	mark entry with this tag (default is getlogin() )
-p priority	facility.level pair
-i	log the process ID with each message
-f	log from the contents of the file message or log from stdin

Example code

A developer first configures default settings for message strings using openlog(), and then logs messages to syslog() as needed.

Configure the log file:

openlog("Reactor control subsystem",LOG_PID|LOG_CONS,LOG_LOCAL0);

Send a message:

syslog(LOG_ALERT,"meltdown imminent!");

Configuring syslog

When the operating system is booted, syslogd is brought up as a user space process. syslogd is started early in the start sequence so that subsequent services can use syslog logging if needed. When the syslogd daemon is started, it reads the configuration file /etc/syslog.conf for its initial configuration. One can reconfigure syslogd and cause it to re-read its configuration file by sending it the SIGHUP signal (kill -1 syslogd_pid). The rules in /etc/syslog.conf are set by the system administrator.

Each non-comment line of /etc/syslog.conf is read as a configuration directive.

Priority tags can be one of the following ASCII strings; these strings correspond to the priority tags defined earlier for the syslog(3c) interface:

emerg or panic

alert

crit

error or err

warning or warn

notice

info

debug

Similarly, the facility tag can be one of the following strings, also corresponding to the facility tags defined earlier for syslog(3c):

kern

user

mail

daemon

auth or security

mark

syslog

lp or lpr

local[0-7]

Examples of possible selectors include:

user.error <tab> /var/adm/syslog/syslog.log # non-critical 

*.emerg <tab> /dev/console # emergencies to console

The action field specifies where the message is to be directed. Possible locations include:

file name	can be a regular file, for example a log file
device file	for example, the console at /dev/console
a list of users	write to a user if logged in
*	write to all users currently logged in
@remote-host	forward to syslogd on a remote system

Here are examples of selectors one might use:

*.emerg <tab> /dev/console # might be crashing!

*.emerg <tab> * # let users know too

*.alert <tab> root,fenwick # let root know ALERTS

*.info <tab> /var/adm/syslog/syslog.log # standard log file

The <tab> character (ASCII tab) is important; in HP-UX this separator must be the tab character, and not a blank character or any other whitespace. Starting with the HP-UX 10.x releases, the standard location for logging syslog messages is the file /var/adm/syslog/syslog.log. The log file from the immediate prior bootup or invocation of syslog is preserved in the file /var/adm/syslog/OLDsyslog.log.

Logging to Remote Hosts

One of the most useful features of syslog is that syslogd can write to an Internet-domain socket connected to a socket on a remote host. The local syslogd daemon opens a socket and writes messages to it. A remote syslogd daemon receives the message. This socket is bound to Port 514/udp, which is reserved for syslog in /etc/services. Multiple hosts can be specified, and remote hosts can be chained together. If one is chaining remote hosts together, one should be aware that the logged message will list the system name of the sending system (which may or may not be the originating).

Here is an example of how one would configure syslog for remote logging to the system "admin" in the file /etc/syslog.conf:

# sample entry in the /etc/syslog.conf file

*.error    <tab>   @admin.cup.hp.com   # forward to Admin. station

System-generated syslog messages

Many UNIX subsystems have been converted to log their messages through syslog. One can now review most kernel-generated messages through the syslog logfile. Kernel-generated messages are routed through the character-special device file /dev/klog. Most messages generated by the kernel in bootup and operation are logged to syslog. In fact, most of the information presented on the kernel bootup screens is logged. One can find the same information in the syslog log file that one can find in the kernel message buffer--there is no longer any reason to use the dmesg(1m) command.

Many of the Internet services (such as the inetd server and sendmail) have also been configured to log errors to syslog. The nettl logging service used in network drivers and services is a different logging routine, but it announces its own startup through syslog.

Since syslog operates over TCP/IP networks, networked peripherals can also use syslog services. For example, HP printers that use the JetDirect network interface card can be configured to report error conditions to syslog. This may be a convenient way to detect and report common printer errors.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019