Solaris Service Management Facility (SMF)

• Introduction
• Enabling and Disabling Services
• Stopping, Starting, and Restarting Services
• Observing the Boot Process
• Discovering What's Going Wrong
• Observing Services
• Changing Run Levels
• Enabling, Disabling, and Monitoring Legacy Services
• Adding New Services to inetd.conf

Introduction

Solaris SMF feature is an attempt to upgrade the standard init.d based boot and daemon management scheme typical for Unix System V. They added XML files which makes it more complex, less flexible and more Windows-style, but it does include some useful enhancements.

Still the architecture made this subsystem more complex and less elegant for trivial enhancements in capabilities (and not all of them relevant for servers).  In this sense it is subpar and I think that Solaris engineers could provide a better, more Unix-style solution, but we have what we have. In other words they provided capabilities similar that exist in Linux (service and chkconfig utilities) using significantly more complex and less transparent approach. 

SMF allow:

• Register daemons for startup (like chkconfig in Linux)
• How to start, stop, and restart services (like service in Linux)
• To store relations of the services to one other (which allows it to boot the operating system to a designated run level much more efficiently).

This understanding of dependencies allows in case of failure to restart not only that particular service, but also "dependent" servicest. Which can be achieved in classic setup too by using pseudo comments (like in Linux) or using a "shadow" dependencies file for each service.

With the -m verbose  option, SMF outputs a line for each service that it's starting, which helps to provide a visual feedback for the start-up process. This is a quite welcome addition.

The other change is more controversial and I have impression that it was agents of Microsoft or IBM infiltrated into Sun who lobbies for this change ;-) Instead of grepping through /var/adm/messages  to find an error that for the service that is having a problem now we have am extremely questionable "advantage" of having one log file for each service. They are located in /var/svc/log  for the most part, with pre-single-user milestone service logs in /etc/svc/volatile.

The system can reach the "login" prompt quicker, as only the services dependent on login need to be started before login is started.  Which is a very questionable advantage, as Solaris is a server OS, not a desktop OS, and generally time to login prompt is irrelevant metric for servers, but this feature probably reflects long gone period when Sun has some hopes for the desktop too (at one point they were using Solaris as a desktop system for the staff, which probably was the largest userbase for Solaris 10 desktop ;-)

SMF moves away from Unix simplicity introducing essentially Windows-style solution:

• Services are represented as first-class objects that can be viewed (using the new svcs(1)  command) and managed (using svcadm(1M)  and svccfg(1M)).
• Failed services are automatically restarted in dependency order, whether they failed as the result of administrator error or a software bug, or they were affected by an uncorrectable hardware error.
• More information is available about misconfigured or misbehaving services, including an explanation of why a service isn't running (using svcs -x), as well as individual, persistent log files for each service.
• Problems during the boot process are easier to debug, as boot verbosity can be controlled, service startup messages are logged, and console access is provided more reliably during startup failures.
• Snapshots of service configurations are taken automatically, making it easier to backup, restore, and undo changes to services.
• Services can be enabled and disabled using a supported tool (svcadm(1M)), allowing the changes to persist across upgrades and patches.
• Administrators can securely delegate tasks to non-root users more easily, including the ability to configure, start, stop, or restart services (as described in the smf_security(5)  man page).
• Large systems boot faster by starting services in parallel according to their dependencies.

Compatibility with existing administrative practices has been preserved wherever possible. For example, most site-local and ISV-supplied rc scripts still work as usual.

Enabling and Disabling Services

Releases prior to the Solaris 10 did not have a standard way to permanently disable a service. The typical method used is to rename the relevant rc script to a name that won't get executed, but that change can be overlooked the next time the system is upgraded. Furthermore, inetd-based services are enabled and disabled by a totally different method -- editing a configuration file. Under SMF, both types of services can be configured using the svcadm(1M)  command (similar in functionality to Linux chkconfig command with functionality of service command added), and the changes will persist. Here's a comparison of how to enable and disable some services.

The last argument to svcadm  in these examples is the Fault Managed Resource Identifier (FMRI) of the service.

Note that svcadm  should only be used for SMF services -- legacy rc script-controlled services work the same as in past releases.

Stopping, Starting, and Restarting Services

Traditionally, services have been started by an rc script run at boot, run with the argument start. Some rc scripts provide a stop  option, and a few also allow restart. In SMF, these tasks are all accomplished with the svcadm(1M)  command, as shown in the following table.

The -t  option to svcadm enable  and svcadm disable  indicates that the requested action should be temporary -- it will not affect whether the service is started the next time that the system boots. This is in contrast to the Enabling and Disabling Services example. for example to enable sshd permanently and to start sshd daemon you can use the command:

svcadm enable network/ssh:default

As with the enabling and disabling of services, svcadm  should not be used to control rc script-controlled services; they continue to work the same as in past releases.

Observing the Boot Process

As mentioned in the Notable Changes section of the QuickStart guide, the boot process is much quieter by default than in previous releases of Solaris. This was done to reduce the amount of uninformative "chatter" that might obscure any real problems that might occur during boot.

Some new boot options have been added to control the verbosity of boot. One that you may find particularly useful is -m verbose, which prints a line of information when each service attempts to start up. This is similar to the default boot mode for some other UNIX-based and UNIX-like operating systems. Verbose boot looks like this:

boot -m verbose 

Rebooting with command: boot -m verbose
Boot device: /pci@1c,600000/scsi@2/disk@0,0:a  File and args: -m verbose
SunOS Release 5.10 Version Generic 64-bit
Copyright 1983-2004 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
[ network/pfil:default starting (pfil) ]
[ network/loopback:default starting (Loopback network interface) ]
[ system/filesystem/root:default starting (Root filesystem mount) ]
Oct 18 13:53:02/13: system start time was Mon Oct 18 13:52:57 2004
[ network/physical:default starting (Physical network interfaces) ]
[ system/filesystem/usr:default starting (/usr and / mounted read/write) ]
    ( more service messages elided )
[ system/filesystem/local:default starting (Local filesystem mounts) ]
[ network/ntp:default starting (network time protocol (NTP)) ]
[ system/utmp:default starting (utmpx monitoring) ]
[ system/filesystem/local:default starting (Local filesystem mounts) ]
[ system/console-login:default starting (Console login) ]

demobox console login: checking ufs filesystems
/dev/rdsk/c0t0d0s7: is logging.
Oct 18 13:53:14/50: system/system-log:default starting
Oct 18 13:53:14/51: network/inetd:default starting
Oct 18 13:53:14/52: system/cron:default starting
    ( more service messages elided )

The order of the service start messages may change from boot to boot, because SMF starts services in parallel according to their dependency relationships.

If a service fails to start successfully, warning messages will be printed in addition to the start message. Here's an example where the NTP service failed to start up:

[ system/filesystem/local:default starting (Local filesystem mounts) ]
[ network/ntp:default starting (network time protocol (NTP)) ]
Oct 25 13:58:42/49 ERROR: svc:/network/ntp:default: 
     Method "/lib/svc/method/xntp" failed with exit status 96.
Oct 25 13:58:42 svc.startd[4]: svc:/network/ntp:default: 
     Method "/lib/svc/method/xntp" failed with exit status 96.
[ network/ntp:default misconfigured (see 'svcs -x' for details) ]
[ system/utmp:default starting (utmpx monitoring) ]
    ( more service messages elided )

The first two error messages would appear during both normal boot and verbose boot; the last one (network/ntp:default misconfigured ...) would only appear during verbose boot.

Discovering What's Going Wrong

The Solaris has not had a comprehensive place to look for problems with system services. Some solutions exist to help catch and diagnose these problems, ranging from coreadm(1M)  logging to site-specific monitoring scripts to comprehensive products such as Sun Cluster.

The new svcs(1)  command includes an "explain" option (svcs -x), which prints out detailed, solution-driven messages about the services that are not running. svcs -x  shows when and why the service failed, provides pointers to more information about the problem, and lists what other services are affected by this problem.

Let's continue with the example of the NTP service failing to start up:

svcs -x 
svc:/network/ntp:default (Network Time Protocol (NTP).)
 State: maintenance since Mon Oct 18 13:58:42 2004
Reason: Start method exited with $SMF_EXIT_ERR_CONFIG.
   See: http://sun.com/msg/SMF-8000-KS
   See: ntpq(1M)
   See: ntpdate(1M)
   See: xntpd(1M)
Impact: 0 services are not running.

The NTP service has been placed into maintenance mode because the startup script indicated a problem with the service's configuration. Further information about the service failure is available in the service's log file in the /var/svc/log  directory (or the /etc/svc/volatile  directory). The log file name is based off the short form of the FMRI, with instances of "/" replaced by "-". So the log file for the svc:/network/ntp:default  service is /var/svc/log/network-ntp:default.log. This log file quickly led to the conclusion that the NTP daemon's configuration file, /etc/inet/ntp.conf, had been removed.

Another example shows SMF's ability to track dependencies and point out problems relating to disabled services. We use the -v  option in this example to see the list of impacted services.

svcs -x -vn
svc:/application/print/server:default (LP Print Service)
 State: disabled since Mon Oct 18 16:17:27 2004
Reason: Disabled by an administrator.
   See: http://sun.com/msg/SMF-8000-05
   See: man -M /usr/share/man -s 1M lpsched
Impact: 1 service is not running:
        svc:/application/print/rfc1179:default

Here, the application/print/server:default  service has been explicitly disabled, but another service that depended on it (application/print/rfc1179:default) was not disabled. So the disabling of the first service has kept the second one from running.

Observing Services

In earlier versions of Solaris, the only way to see what services were available was to use the ps(1)  command and list all the active processes on the system, and then look around for the names of processes that matched the names of service applications. Unfortunately, it's very difficult to track things this way since most systems have many processes, and new services are introduced with each new version of Solaris and when other software packages are added. To further complicate the situation, many modern services are no longer implemented as single processes. Some services are implemented as collections of processes, multithreaded processes, or both simultaneously.

The new svcs(1)  command makes it much easier to observe the status of a system service. The -p  option shows all the processes associated with a service:

svcs -p network/smtp:sendmail 
STATE          STIME    FMRI
online         18:20:30 svc:/network/smtp:sendmail
               18:20:30      655 sendmail
               18:20:30      657 sendmail

% ps -fp 655,657
     UID   PID  PPID   C    STIME TTY   TIME CMD
    root   655     1   0 18:20:30 ?     0:01 /usr/lib/sendmail -bd -q15m
   smmsp   657     1   0 18:20:30 ?     0:00 /usr/lib/sendmail -Ac -q15m

The -d  option shows what other services this service depends on, and the -D  option shows what other services depend on this service:

svcs -d network/smtp:sendmail 
STATE          STIME    FMRI
online         18:20:14 svc:/system/identity:domain
online         18:20:26 svc:/network/service:default
online         18:20:27 svc:/system/filesystem/local:default
online         18:20:27 svc:/milestone/name-services:default
online         18:20:27 svc:/system/system-log:default
online         18:20:30 svc:/system/filesystem/autofs:default
 svcs -D network/smtp:sendmail 
STATE          STIME    FMRI
online         18:20:32 svc:/milestone/multi-user:default

We can see that sendmail requires networking, local file systems, name services, the syslog daemon, and the automount daemon to be running before it will run, and sendmail itself must be running before the multi-user milestone can be reached. The service start times (the STIME  column) illustrate that these dependencies have been followed.

Changing Run Levels

SMF has introduced the concept of milestones, which supplant the traditional notion of run levels. Run levels provide a basic description of the set of services running on the machine, traditionally grouped as the services necessary for one user to log in on the machine console (run level S), and for multiple users to log in to the machine (run levels 2 and 3). These system states are represented in SMF as milestones, which are stable services that represent a group of other services. svcs -d  can be used to see what services must be running before a milestone is reached.

svcadm(1M)  is now the preferred method of setting the system's default run level. This is done with the milestone  subcommand and the FMRI of a valid milestone, as seen in Table 3.

The -d  option indicates that the default milestone should be set to the named FMRI. Without the -d  option, svcadm milestone  transitions the system to the named milestone immediately.

The boot process has been updated to be aware of milestones. In addition to the traditional boot -s  (boot into single-user mode), you now have boot -m milestone=<milestone>  to boot to the named milestone. <milestone>  can be single-user, multi-user, or multi-user-server, as well as the special milestones all  (all enabled services online) and none  (no services at all). The none  milestone can be very useful in repairing systems that have failures early in the boot process.

Booting to the single-user milestone (with -m milestone=single-user) is slightly different than using the old boot -s. When the system is explicitly booted to a milestone, exiting the console administrative shell will not transition the system to multi-user mode, as boot -s  does. To move to multi-user mode after boot -m milestone=single-user, use the command svcadm milestone milestone/multi-user-server:default.

Enabling, Disabling, and Monitoring Legacy Services

Services that are started by traditional rc scripts (referred to as legacy services) will generally continue to work as they always have. They will show up in the output of svcs(1), with an FMRI based on the path name of their rc script, but they cannot be controlled by svcadm(1M). They should be stopped and started by running the rc script directly.

The rc scripts may not run at exactly the same point in boot as they had in earlier versions of Solaris. In particular, problems may arise for scripts that depend on running before certain rc scripts provided in the Solaris. The vast majority of scripts should continue to work without any trouble, though.

Adding New Services to inetd.conf

The Internet services daemon, inetd(1M), has been rewritten as part of SMF. It stores all of its configuration data in the SMF database, rather than /etc/inet/inetd.conf, allowing the SMF tools to be used to control and observe inetd-based services.

Most inetd-based services that ship with the Solaris will no longer have entries in inetd.conf.

To provide compatibility for services which haven't converted to SMF, entries can still be added to inetd.conf  using the same syntax as always, and the new inetconv(1M)  command will convert the new services to SMF services. inetconv  should always be run after editing /etc/inet/inetd.conf; it can be run without any arguments.

Recommended Links

Softpanorama Recommended

Top articles

Sites

• QuickStart guide,
• BigAdmin Feature Article Using Service Management Facility (SMF) in the Solaris 10 A Quick Example
• SMF and Predictive Self-Healing.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 12, 2019