Solaris implementation of sar

sar is native activity reporter package included in the Solaris operating environment. It can be enabled by uncommenting the appropriate lines in the system crontab.  This package consists of main utility sar and  data collector sadc and two scripts sa1, and sa2.

• sadc, a utility that also is part of the system activity reporter, collects system activity data and saves it in a binary format—one file for each 24-hour period. This data includes information on CPU utilization, buffer usage, disk and tape I/O activity, TTY device activity, switching and system-call activity, file access, queue activity, interprocess communications, and paging. The information saved is very much like the information you see displayed with the vmstat and iostat commands.

When run, sadc creates binary files in the /usr/adm/sa directory. Each file is named sa<dd>, where <dd> is the current date. The syntax for the sadc command is as follows:

/usr/lib/sa/sadc [t n] <outputfile>

t is the interval (seconds) between samples, and n is the number of samples to take. The value of t should be greater than five seconds; otherwise, the activity of sadc itself might affect the sample. sadc then writes, in binary format, to the file named <outputfile> or to standard output if a filename is not specified. If t and n are omitted, a special file is written once.

The sadc command should be run first at system boot time to record the statistics from when the counters are reset to zero. To make sure sadc is run at boot time, the /etc/init.d/perf file must contain a command line that writes a record to the daily data file.

The command entry has the following format:

su sys -c "/usr/lib/sa/sadc /usr/adm/sa/sa´date +%d´"

This entry is already in the /etc/init.d/perf file, but it needs to be uncommented.

Typically, the system administrator sets up sadc to run at boot time when the system enters multiuser mode and then periodically—usually once each hour. To generate periodic records, you need to run sadc regularly. The simplest way to do this is by putting a line into the /var/spool/cron/crontabs/ sys file, which calls the shell script, sa1. This script invokes sadc and writes to the daily data files, /var/adm/sa/sa<dd>. The sa1 script gets installed with the other sar packages and has the following syntax:

/usr/lib/sa/sa1 [t n]

As with sadc, the arguments t and n cause records to be written n times at an interval of t seconds. If these arguments are omitted, the records are written only one time.

The sar command is used to report what has been collected in the daily activity files created by sadc. sadc creates binary files, and the only way to read this data is using the sar utility.

In addition, the sar command can be used to gather system activity data from the command line to look at performance either over different time slices or over a constricted period of time.

The syntax for the sar command is as follows:

sar [-aAbcdgkmpqruvwy] [-o <outputfile>] [t n ]

The options"

• a - Checks file access operations
• b - Checks buffer activity
• c - Checks system calls
• d - Checks activity for each block device
• g - Checks page-out and memory freeing
• k - Checks kernel memory allocation
• m - Checks interprocess communication
• p - Checks swap and dispatch activity
• q - Checks queue activity
• r - Checks unused memory
• u - Checks CPU utilization
• nv - Checks system table status
• w - Checks swapping and switching volume
• y - Checks terminal activity
• A - Reports overall system performance (same as entering all options)

If no option is used, it is equivalent to calling the command with the -u option.

For example:

sar -A -o outfile 5 500

This will display all system activity data every 5 seconds and will gather 500 samples. The file named outfile is the output file where the data will get stored. The data is stored in binary format but can be read by using the -f option, as follows:

sar -f outfile

The following information is displayed:

SunOS ultra5 5.9 sun4u  05/14/2002

15:44:26  %usr  %sys  %wio  %idle
15:44:31    1    2    0   97
15:44:36    1    2    0   96
15:44:41    9    2    9   80
15:44:46    2    3    0   95
15:44:51    0    1    0   99
15:44:56    3    2    0   96
15:45:01    0    0    0   100
15:45:06   15    0    0   85
15:45:11    2    0    0   97
15:45:16    3    2    0   95
15:45:21    1    1    0   98
15:45:26    2    2    0   97
15:45:31    0    0    0   100
15:45:36    0    0    0   100
15:45:41    8    1    0   92
15:45:46    1    4    0   95
15:45:51    2    3    0   96
15:45:56    2    0    0   98
15:46:01    0    0    0   100
15:46:06   15    0    0   85

Average    3    1    0   95

To start collecting system activity data with sar, you need to edit the /etc/ init.d/perf file and create an entry in the crontab file. Here are the steps:

1. Edit the /etc/init.d/perf file and follow instructions in that file to uncomment lines that enable system activity gathering.
2. The sadc command writes a special record that marks the time when the counters are reset to zero (boot time). The sadc output is put into the file sa<dd> (where dd is the current date), which acts as the daily system activity record.
3. Edit the /var/spool/cron/crontabs/sys file (the system crontab file) and uncomment the following lines:

0 * * * 0-6 /usr/lib/sa/sa1
20,40 8-17 * * 1-5 /usr/lib/sa/sa1
5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A

The first entry writes a record to /var/adm/sa/sa<dd> on the hour, every hour, seven days a week.

The second entry writes a record to /var/adm/sa/sa<dd> twice each hour during peak working hours: at 20 minutes and 40 minutes past the hour, from 8 a.m. to 5 p.m., Monday through Friday.

Thus, these two crontab entries cause a record to be written to /var/adm/sa/sa<dd> every 20 minutes from 8 a.m. to 5 p.m., Monday through Friday, and every hour on the hour otherwise. You can change these defaults to meet your needs.

The third entry runs the shell script named sa2 at 6:05 p.m., Monday through Friday. The sa2 script writes reports from the binary data.

The shell script sa2, a variant of sar, writes a daily report in the file /var/ adm/sa/sar<dd>. The report will summarize hourly activities for the day.

In his article Easy system monitoring with SAR (Feb 28, 2006, IBM DeveloperWorks) Sean Walberg gave several typical example on how to extract system performance information using sar: 

Extracting useful information

Data is being collected, but it must be queried to be useful. Running the sar command without options generates basic statistics about CPU usage for the current day. Listing 2 shows the output of sar without any parameters. (You might see different column names depending on the platform. In some UNIX flavors, sadc collects more or less data based on what's available.) The examples here are from Sun Solaris 10; whatever platform you're using will be similar, but might have slightly different column names.

Listing 2. Default output of sar (showing CPU usage

-bash-3.00$ sar

SunOS unknown 5.10 Generic_118822-23 sun4u    01/20/2006

00:00:01    %usr    %sys    %wio   %idle
00:10:00       0       0       0     100
. cut ...
09:30:00       4      47       0      49

Average        0       1       0      98

Each line in the output of sar is a single measurement, with the timestamp in the left-most column. The other columns hold the data. (These columns vary depending on the command-line arguments you use.) In Listing 2, the CPU usage is broken into four categories:

• %usr: The percentage of time the CPU is spending on user processes, such as applications, shell scripts, or interacting with the user.
• %sys: The percentage of time the CPU is spending executing kernel tasks. In this example, the number is high, because I was pulling data from the kernel's random number generator.
• %wio: The percentage of time the CPU is waiting for input or output from a block device, such as a disk.
• %idle: The percentage of time the CPU isn't doing anything useful.

The last line is an average of all the datapoints. However, because most systems experience busy periods followed by idle periods, the average doesn't tell the entire story.

Watching disk activity

Disk activity is also monitored. High disk usage means that there will be a greater chance that an application requesting data from disk will block (pause) until the disk is ready for that process. The solution typically involves splitting file systems across disks or arrays; however, the first step is to know that you have a problem.

The output of sar -d shows various disk-related statistics for one measurement period. For the sake of brevity, Listing 3 shows only hard disk drive activity.

Listing 3. Output of sar -d (showing disk activity)
 

$ sar -d SunOS unknown 5.10 Generic_118822-23 sun4u 01/22/2006 00:00:01 device %busy avque r+w/s blks/s avwait avserv . cut ... 14:00:02 dad0 31 0.6 78 16102 1.9 5.3 dad0,c 0 0.0 0 0 0.0 0.0 dad0,h 31 0.6 78 16102 1.9 5.3 dad1 0 0.0 0 1 1.6 1.3 dad1,a 0 0.0 0 1 1.6 1.3 dad1,b 0 0.0 0 0 0.0 0.0 dad1,c 0 0.0 0 0 0.0 0.0

As in the previous example, the time is along the left. The other columns are as follows:

• device: This is the disk, or disk partition, being measured. In Sun Solaris, you must translate this disk into a physical disk by looking up the reported name in /etc/path_to_inst, and then cross-reference that information to the entries in /dev/dsk. In Linux®, the major and minor numbers of the disk device are used.
• %busy: This is the percentage of time the device is being read from or written to.
• avque: This is the average depth of the queue that is used to serialize disk activity. The higher the avque value, the more blocking is occurring.
• r+w/s, blks/s: This is disk activity per second in terms of read or write operations and disk blocks, respectively.
• avwait: This is the average time (in milliseconds) that a disk read or write operation waits before it is performed.
• avserv: This is the average time (in milliseconds) that a disk read or write operation takes to execute.

Some of these numbers, such as avwait and avserv values, correlate directly into user experience. High wait times on the disk likely point to several people contending for the disk, which should be confirmed with high avque numbers. High avserv values point to slow disks.

Other metrics

Many other items are collected, with corresponding arguments to view them:

• The -b argument shows information on buffers and the efficiency of using a buffer versus having to go to disk.
• The -c argument shows system calls broken down into some of the popular calls, such as fork(), exec(), read(), and write(). High process creation can lead to poor performance and is a sign that you might need to move some applications to another computer.
• The -g, -p, and -w arguments show paging (swapping) activity. High paging is a sign of memory starvation. In particular, the -w argument shows the number of process switches: A high number can mean too many things are running on the computer, which is spending more time switching than working.
• The -q argument shows the size of the run queue, which is the same as the load average for the time.
• The -r argument shows free memory and swap space over time.

Each UNIX flavor implements its own set of measurements and command-line arguments for sar. Those I've shown are common and represent the elements that I find more useful.

Timing is everything

The examples thus far have shown the current day's data, which has its uses, but it also has two problems:

• You're interested in an hour of data, but you get the whole day.
• You need to go back to a different day.

As you saw earlier, sa1 saves the data in a different file for each day. Looking at the sa1 script itself tells you which directory is used; in the case of Sun Solaris 10, it is in /var/adm/sa. Several files reside in this directory, starting with either "sa" or "sar" followed by a number. The number represents the day of the month, with the files beginning with "sar" being text dumps of the data for that day (created by the nightly run of sa2) and the files beginning with "sa" holding the binary version. Indeed, the file containing the current date is the file that is being read from when you launch sar.

Specifying -f to the sar command selects the file to read from. If today were the 23rd day of the month, I could look at yesterday's data by reading from sa22 with the command sar -f /var/adm/sa/sa22. You can also pass the other arguments I showed you to access different types of data.

The second thing you can do to narrow the scope of the query is to specify the time by using the -s and -e arguments (think start and end). Note that -s is not inclusive, so you must subtract an extra ten minutes from the chosen start time. Continuing with the previous example, Listing 4 shows swap file usage and the run queue for the 22nd from 2:30 p.m. to 3:00 p.m.

Listing 4. A complex sar query specifying date, time, and multiple data sets
 

# sar -f /var/adm/sa/sa22 -s 14:20 -e 15:00 -w -q -i 4 SunOS unknown 5.10 Generic_118822-23 sun4u 01/22/2006 14:20:00 swpin/s bswin/s swpot/s bswot/s pswch/s 14:30:00 0.00 0.0 0.00 0.0 140 14:40:01 0.00 0.0 0.00 0.0 144 14:50:01 0.00 0.0 0.00 0.0 140 15:00:00 0.00 0.0 0.00 0.0 139 Average 0.00 0.0 0.00 0.0 140 14:20:00 runq-sz %runocc swpq-sz %swpocc 14:30:00 10.5 100 0.0 0 14:40:01 10.5 100 0.0 0 14:50:01 10.4 100 0.0 0 15:00:00 10.5 100 0.0 0 Average 10.5 100 0.0 0

Making sense of it all

A brief look at Listing 4 shows that swap activity was NIL, approximately 140 process switches per second occurred, and the load average was slightly more than ten. Assuming that you were investigating a claim of poor performance at the time, what does this tell you?

• Whatever process is running isn't memory intensive, because you don't see swapping.
• Chances are that this problem is caused by a long-running set of processes, because the run queue and process switches are relatively consistent. Had they not been, you could suspect application-level problems, such as a busy Web server.
• Knowing that the output of Listing 3 shows part of the same time period, you can see that one of the disks was being used heavily (31 percent according to sar -b, but also 16,000 blocks per second). This disk is the home directory partition; depending on what the user was trying to do, he or she might have experienced slow responses.

A quick look at the CPU usage for the time period shows that the system took up approximately 80 percent of the CPU; the rest was consumed by user tasks. As the systems administrator, you can use this information in three ways:

• Go back over previous days' logs. In this case, I found that the problem started at 1:00 p.m. and ended the next morning.
• Try to correlate the activity to any cron jobs that might have been started that day.
• Try to find a trend. Looking at data from a couple of other days, I saw that the performance was normal, which isn't indicative of a system that has reached its limits.

In this case, the problem seemed to be isolated, and for good reason -- I was intentionally running the disks with shell scripts to create some interesting sar reports! However, had a trend appeared, such as busy home drives during working hours, it would have been a call to do something about the problem. Possible solutions range from splitting home directories off to other disks, installing faster disks, or moving to something like Network Attached Storage (NAS).

Conclusion

Obtaining qualitative data about your system at periodic intervals is an effective way of finding performance bottlenecks and determining whether further action is needed. SAR and related utilities do just this -- snapshots are taken every ten minutes and a front end allows you to access this data. Though tactical in nature, a wealth of information is provided that enables systems administrators to discover just what aspect of the system is suffering and whether it requires further investigation.

NEWS CONTENTS

• 20060228 : Easy system monitoring with SAR by Sean Walberg ( Easy system monitoring with SAR, Feb 28, 2006 )
• 20060228 : Solaris 9 System Monitoring and Tuning Fixing a Performance Problem ( Solaris 9 System Monitoring and Tuning Fixing a Performance Problem, )
• 20060228 : Table 19.3 Options for the sar Command ( )
• 200102 : Computer Performance Monitoring ( Computer Performance Monitoring, )

Old News ;-)

[Feb 28, 2006] Easy system monitoring with SAR by Sean Walberg

www.ibm.com/developerworks

Extracting useful information

Data is being collected, but it must be queried to be useful. Running the sar command without options generates basic statistics about CPU usage for the current day. Listing 2 shows the output of sar without any parameters. (You might see different column names depending on the platform. In some UNIX flavors, sadc collects more or less data based on what's available.) The examples here are from Sun Solaris 10; whatever platform you're using will be similar, but might have slightly different column names.

Listing 2. Default output of sar (showing CPU usage

-bash-3.00$ sar

SunOS unknown 5.10 Generic_118822-23 sun4u    01/20/2006

00:00:01    %usr    %sys    %wio   %idle
00:10:00       0       0       0     100
. cut ...
09:30:00       4      47       0      49

Average        0       1       0      98

Each line in the output of sar is a single measurement, with the timestamp in the left-most column. The other columns hold the data. (These columns vary depending on the command-line arguments you use.) In Listing 2, the CPU usage is broken into four categories:

• %usr: The percentage of time the CPU is spending on user processes, such as applications, shell scripts, or interacting with the user.
• %sys: The percentage of time the CPU is spending executing kernel tasks. In this example, the number is high, because I was pulling data from the kernel's random number generator.
• %wio: The percentage of time the CPU is waiting for input or output from a block device, such as a disk.
• %idle: The percentage of time the CPU isn't doing anything useful.

The last line is an average of all the datapoints. However, because most systems experience busy periods followed by idle periods, the average doesn't tell the entire story.

Watching disk activity

Disk activity is also monitored. High disk usage means that there will be a greater chance that an application requesting data from disk will block (pause) until the disk is ready for that process. The solution typically involves splitting file systems across disks or arrays; however, the first step is to know that you have a problem.

The output of sar -d shows various disk-related statistics for one measurement period. For the sake of brevity, Listing 3 shows only hard disk drive activity.

Listing 3. Output of sar -d (showing disk activity)

$ sar -d

SunOS unknown 5.10 Generic_118822-23 sun4u    01/22/2006

00:00:01   device       %busy   avque   r+w/s  blks/s  avwait  avserv
. cut ...
14:00:02   dad0             31     0.6      78   16102     1.9     5.3
           dad0,c            0     0.0       0       0     0.0     0.0
           dad0,h           31     0.6      78   16102     1.9     5.3
           dad1              0     0.0       0       1     1.6     1.3
           dad1,a            0     0.0       0       1     1.6     1.3
           dad1,b            0     0.0       0       0     0.0     0.0
           dad1,c            0     0.0       0       0     0.0     0.0

As in the previous example, the time is along the left. The other columns are as follows:

• device: This is the disk, or disk partition, being measured. In Sun Solaris, you must translate this disk into a physical disk by looking up the reported name in /etc/path_to_inst, and then cross-reference that information to the entries in /dev/dsk. In Linux®, the major and minor numbers of the disk device are used.
• %busy: This is the percentage of time the device is being read from or written to.
• avque: This is the average depth of the queue that is used to serialize disk activity. The higher the avque value, the more blocking is occurring.
• r+w/s, blks/s: This is disk activity per second in terms of read or write operations and disk blocks, respectively.
• avwait: This is the average time (in milliseconds) that a disk read or write operation waits before it is performed.
• avserv: This is the average time (in milliseconds) that a disk read or write operation takes to execute.

Some of these numbers, such as avwait and avserv values, correlate directly into user experience. High wait times on the disk likely point to several people contending for the disk, which should be confirmed with high avque numbers. High avserv values point to slow disks.

Other metrics

Many other items are collected, with corresponding arguments to view them:

• The -b argument shows information on buffers and the efficiency of using a buffer versus having to go to disk.
• The -c argument shows system calls broken down into some of the popular calls, such as fork(), exec(), read(), and write(). High process creation can lead to poor performance and is a sign that you might need to move some applications to another computer.
• The -g, -p, and -w arguments show paging (swapping) activity. High paging is a sign of memory starvation. In particular, the -w argument shows the number of process switches: A high number can mean too many things are running on the computer, which is spending more time switching than working.
• The -q argument shows the size of the run queue, which is the same as the load average for the time.
• The -r argument shows free memory and swap space over time.

Each UNIX flavor implements its own set of measurements and command-line arguments for sar. Those I've shown are common and represent the elements that I find more useful.

Solaris 9 System Monitoring and Tuning Fixing a Performance Problem

Gathering system activity data from vmstat and iostat can be a time-consuming job. You need to automate the gathering of system activity data, and the tool to use is sar. sar refers to the system activity reporter and is used to gather performance data from all the major components of the system. sar comes standard on most UNIX systems, including Solaris. sar can be enabled by uncommenting the appropriate lines in the system crontab and is described later in this section.

sar is part of the system activity reporter package that is included in the Solaris 9 operating environment. This package consists of three commands that are involved in automatic system activity data collection: sadc, sa1, and sa2.

sadc, a utility that also is part of the system activity reporter, collects system activity data and saves it in a binary format-one file for each 24-hour period. This data includes information on CPU utilization, buffer usage, disk and tape I/O activity, TTY device activity, switching and system-call activity, file access, queue activity, interprocess communications, and paging. The information saved is very much like the information you see displayed with the vmstat and iostat commands.

When run, sadc creates binary files in the /usr/adm/sa directory. Each file is named sa<d> n is the number of samples to take. The value of t should be greater than five seconds; otherwise, the activity of sadc itself might affect the sample. sadc then writes, in binary format, to the file named <outputfile> or to standard output if a filename is not specified. If t and n are omitted, a special file is written once.

The sadc command should be run first at system boot time to record the statistics from when the counters are reset to zero. To make sure sadc is run at boot time, the /etc/init.d/perf file must contain a command line that writes a record to the daily data file.

The command entry has the following format:

su sys -c "/usr/lib/sa/sadc /usr/adm/sa/sa´date +%d´"

This entry is already in the /etc/init.d/perf file, but it needs to be uncommented.

Typically, the system administrator sets up sadc to run at boot time when the system enters multiuser mode and then periodically-usually once each hour. To generate periodic records, you need to run sadc regularly. The simplest way to do this is by putting a line into the /var/spool/cron/crontabs/ sys file, which calls the shell script, sa1. This script invokes sadc and writes to the daily data files, /var/adm/sa/sa<dd>. The sa1 script gets installed with the other sar packages and has the following syntax:

/usr/lib/sa/sa1 [t n]

As with sadc, the arguments t and n cause records to be written n times at an interval of t seconds. If these arguments are omitted, the records are written only one time.

The sar command is used to report what has been collected in the daily activity files created by sadc. sadc creates binary files, and the only way to read this data is using the sar utility.

In addition, the sar command can be used to gather system activity data from the command line to look at performance either over different time slices or over a constricted period of time.

The syntax for the sar command is as follows:

sar [-aAbcdgkmpqruvwy] [-o <outputfile>] [t n ]

The options to the sar command are described in Table 19.3.

Table 19.3 Options for the sar Command

Option

Action a

Checks file access operations b

Checks buffer activity c

Checks system calls d

Checks activity for each block device g

Checks page-out and memory freeing k

Checks kernel memory allocation m

Checks interprocess communication p

Checks swap and dispatch activity q

Checks queue activity r

Checks unused memory u

Checks CPU utilization nv

Checks system table status w

Checks swapping and switching volume y

Checks terminal activity A

Reports overall system performance (same as entering all options)

NOTE

If no option is used, it is equivalent to calling the command with the -u option.

For example:

sar -A -o outfile 5 500

This will display all system activity data every 5 seconds and will gather 500 samples. The file named outfile is the output file where the data will get stored. The data is stored in binary format but can be read by using the -f option, as follows:

sar –f outfile

The following information is displayed:

SunOS ultra5 5.9 sun4u  05/14/2002

15:44:26  %usr  %sys  %wio  %idle
15:44:31    1    2    0   97
15:44:36    1    2    0   96
15:44:41    9    2    9   80
15:44:46    2    3    0   95
15:44:51    0    1    0   99
15:44:56    3    2    0   96
15:45:01    0    0    0   100
15:45:06   15    0    0   85
15:45:11    2    0    0   97
15:45:16    3    2    0   95
15:45:21    1    1    0   98
15:45:26    2    2    0   97
15:45:31    0    0    0   100
15:45:36    0    0    0   100
15:45:41    8    1    0   92
15:45:46    1    4    0   95
15:45:51    2    3    0   96
15:45:56    2    0    0   98
15:46:01    0    0    0   100
15:46:06   15    0    0   85

Average    3    1    0   95

Setting Up sar

To start collecting system activity data with sar, you need to edit the /etc/ init.d/perf file and create an entry in the crontab file. Here are the steps:

1. Become superuser.
2. Edit the /etc/init.d/perf file and follow instructions in that file to uncomment lines that enable system activity gathering.
3. The sadc command writes a special record that marks the time when the counters are reset to zero (boot time). The sadc output is put into the file sa<dd> (where dd is the current date), which acts as the daily system activity record.
4. 3Edit the /var/spool/cron/crontabs/sys file (the system crontab file) and uncomment the following lines:

0 * * * 0-6 /usr/lib/sa/sa1
20,40 8-17 * * 1-5 /usr/lib/sa/sa1
5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A

The first entry writes a record to /var/adm/sa/sa<dd> on the hour, every hour, seven days a week.

The second entry writes a record to /var/adm/sa/sa<dd> twice each hour during peak working hours: at 20 minutes and 40 minutes past the hour, from 8 a.m. to 5 p.m., Monday through Friday.

Thus, these two crontab entries cause a record to be written to /var/adm/sa/sa<dd> every 20 minutes from 8 a.m. to 5 p.m., Monday through Friday, and every hour on the hour otherwise. You can change these defaults to meet your needs.

The third entry runs the shell script named sa2 at 6:05 p.m., Monday through Friday. The sa2 script writes reports from the binary data.

The shell script sa2, a variant of sar, writes a daily report in the file /var/ adm/sa/sar<dd>. The report will summarize hourly activities for the day.

SyMON 3.0 was developed by the server division of Sun to provide a graphical user interface to its server products. It is deliberately very hardware specific, and it covers functionality that generic tools from other vendors do not address. In particular, it contains detailed knowledge of the configuration of a system and identifies low-level faults (such as correctable memory and disk errors) or high temperature levels that might lead to downtime if they are not identified and acted on. SyMON has a configuration browser that contains full-color images of system components as well as a simple hierarchy view.

The GUI monitors only a single system at a time but is a distributed application. A data collection process is all that runs on the server. An event monitor process runs on a separate machine and looks for problems. In the event of system downtime, the event monitor can raise the alarm even if the GUI is not in use. The Motif GUI can be installed on a separate desktop system. As a performance management tool, SyMON is equivalent to HP's GlancePlus product in that it does not store long-term data (a rolling two-hour history is available), but it does graph all the usual performance data. Its configurable alert capability has many predefined rules and can pass SNMP traps to a network management package; it also has an event log viewer.

SyMON can monitor any of Sun's UltraSPARC-based systems. It is supported only on server configurations, but because the Ultra 1 and Ultra 2 workstations are the basis for the Enterprise 150 and Enterprise 2, it works on them just as well. (The graphical view of an E150 appears if you use it on an Ultra 1, however.)

Solaris 9 System Monitoring and Tuning sar

Computer Performance Monitoring

Solaris ps presents per-process stats.

Solaris prstat presents thread-level microstate accounting (with high-resolution time stamps) and per-project stats for resource management.

sar -v [entries/size for each table, evaluated once at sampling point, not available in SiteScope]:

proc-sz = number of process entries (proc structures) that are currently being used, or allocated in the kernel.

inod-sz = total number of inodes in memory versus the maximum number of inodes that are allocated in the kernel. This number is not a strict high water mark. The number can overflow.
file-sz = size of the open system file table. The sz is given as 0, since space is allocated dynamically for the file table.

ov = overflows that occur between sampling points for each table. The number of shared memory record table entries currently being used or allocated in the kernel. The sz is given as 0 because space is allocated dynamically for the shared memory record table.

lock-sz = number of shared memory record table entries currently being used or allocated in the kernel. The sz is given as 0 because space is allocated dynamically for the shared memory record table.

sar -c [System calls]:

scall/s = All types of system calls per second, which is generally about 30 per second on a system with 4 to 6 users.
sread/s swrit/s = read system calls per second.
fork/s = write system calls per second.
exec/s = exec system calls per second. If exec/s divided by fork/s is greater than three, look for inefficient PATH variables.
rchar/s, wchar/s = characters (bytes) transferred by read and write system calls per second.

sar -m [Message and semaphore activities (for Interprocess Communication)]:

msg/s = message primitives (send and receive operations) per second.
sema/s = sempahore primitives per second.
These figures will usually be zero (0.00), unless you are running applications that use messages or semaphores.

sar -t [translation lookaside buffer (TLB) activities, not available in SiteScope]:

tflt/s = user page table or kernel virtual address translation
faults: address translation not resident in TLB;
rflt/s = page reference faults (valid page in memory, but hardware valid bit disabled to emulate hardware reference bit);
sync/s = TLBs flushes on all processors;
vmwrp/s = syncs caused by clean (with respect to TLB) kernel virtual memory depletion;
flush/s = single processor TLB flushes;
idwrp/s = flushes because TLB ids have been depleted;
idget/s = new TLB ids issued;
idprg/s = tlb ids purged from process;
vmprg/s = individual TLB entries purged.

sar -I [interrupt statistics, not available in SiteScope]:

intr/s = non-vme interrupts per second;
vmeintr/s = vme interrupts per second;

sar -a [File access system routines]:

iget/s = number of requests made for inodes that were not in the directory name look-up cache (DNLC).

namei/s = number of file system path searches per second. If namei does not find a directory name in the DNLC, it calls iget to get the inode for either a file or directory. Hence, most igets are the result of DNLC misses.

dirbk/s = number of directory block reads issued per second.

Recommended Links

Softpanorama Recommended

Top articles

Sites

Easy system monitoring with SAR by Sean Walberg

Feb 28, 2006 | IBM DeveloperWorks

Linux.com CLI Magic Tracking system performance with sar

ITworld.com - UNIX SYSTEM ADMINISTRATION - Introducing SAR

PDF] Solaris Performance monitoring - sar - NCCCS Systems Office Wiki

freshmeat.net Project details for BSDsar -- BSDsar generates a history of usage on a FreeBSD machine. It logs data such as CPU usage, disk activity, network bandwidth usage and activity, NFS information, memory, and swap. It is similar to atsar (for Linux) and sar (for Solaris).

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019