Back to basics -- Softpanorama sysadmin utilities

• Introduction
• Does continuing this project make any sense?
• The list of utilities I plan to publish

Introduction

This project is about publishing of a small set of sysadmin utilities that might help to administer Unix server in a "classic" way: using command line tools, pipes and, if necessary, bash as a glue that connects small scripts (Written in Perl or Python or Ruby), or programs (for example written in GOlang) each of which perform some limited, well defined function.  This is an alternative of adopting yet another complex system with yet another DSL (domain specific language) for each and every problem. 

Actually Unix created a new style of computing, a new way of thinking of how to attack complex problems, especially sysadmin problems. This style was essentially the first successful component model in programming. And despite its age it still holds it own.  As Frederick P. Brooks Jr (another computer pioneer who early recognized the importance of pipes) noted, the creators of Unix "...attacked the accidental difficulties that result from using individual programs together, by providing integrated libraries, unified file formats, and pipes and filters."

By sticking to a single integration language (bash), this approach somewhat differs from the approach based on scripting written in the DSL for particular configuration management system, be it Ansible, Puppet or something else. All of them reintroduced on a new "parallelized"  level ideas of IBM JCL into Unix environment -- waterfall execution of steps required for accomplishing given task. Those utilities, of course, can be used as a part of Ansible scripts, when it makes sense. But most often using bash and pdsh ( or Ansible in "ad hoc" mode, cexec, parallel or other similar tool)  is simpler and more straightforward, more modifiable, more easily manageable.

BTW, historically, Unix shell (and IBM REXX) wiped the floor with JCL.  So instead of having, say, Ansible as a superstructure with its own DSL, you use it as a component (in "as hoc" mode) and bash as our DSL.  With all its warts bash as a programming language is superior to Ansible DSL.

The utilities in question were written by me over the years and most of them have a common "compiler style" framework with the special emphasis on providing meaningful diagnostic messages. To that extent they all rely of a set of components that provide the generation of messages somewhat similar in style to the old IBM PL/1 compliers. Verbosity can be regulated via option -v. 

Quality, the level of maturity and usefulness vary. You can view them as a kind of my personal protest against the overcomplexity of the current sysadmin environment. Now there are way too many tools available to make simple tasks complex and complex unsolvable :-)

Generally Linux system administration gradually moves to the "Windows-way" of doing things -- to tools that provide GUI and "blackboxing" of OS API and internals. Many DevOps toys can be viewed as steps in this particular direction. Some new subsystems like systemd also belong to this category. They all have their place but they all add too much complexity and in small companies adoption of them entails significant risks due to the lack of people able to master them all. 

In this sense you can view this collection as a very small contribution to the "back to basics" movement among sysadmins.

Moreover, for small and medium organization adoption of overly complex tools entrain significant risks. If the organization adopted a set of complex tools and a person who known those tool leaves, often the situation deteriorates pretty quickly as there is no cost effective way to replace him/her. Lack of documentation is typical and can bite very hard to the level of partial or complete paralysis.

Sticking to classic Unix capabilities often is a better approach to system administration then the adoption of a bunch of complex shiny tools that no normal person can learn in depth in his life. Drawbacks often cited for "primitive" approaches of managing servers (for example with cloning /etc/passwd /etc/group files using ssh instead of Active directory or some other directory) can often be compensated quite easity (for example with the automatic synchronization of passwd files on demand via ssh form some "etalon" server, see below ; it is also possible decompile and generate useradd command using diff of two passwd/group files). Similarly a lot of collection of data for monitoring can be done using NFS3 and does not require SSH or some proprietary protocol. And analysis of logs is better performed by custom utilities tuned to your particular situation or at least enhanced third party scripts, instead of some complex system. Same is true for backup although here your mileage may vary. Often "bare metal" backup can be done via tar or Rsync and does not require complex tools with additional (and probably less secure then ssh) protocols.

At the same time classic Unix provides unmatched flexibility which some modern approaches considerably diminish squeezing sysadmin into Procrustean bed of badly designed and overly complex solutions.

Initially I published on GitHub three utilities from this set: neatperl, neatbash and saferm . IMHO neatbash fills the "missing link" in creation of simple bash IDE, based on Midnight commander and some editor (say, vim). I thought that sysadmin will appreciate and it will use it. Unfortunately it was by-and-large ignored. And for developers one of main stimulus for further development is the positive feedback loop.  As Fred Brooks notes in 1975 in his groundbreaking book "The Mythical Man-Month: Essays on Software Engineering" the difference in effort required to produce the  utility "for yourself" and its publishable form can be ten times or more. This sacrifices in time and  effort is difficult to commit too if you suspect that this is all  "putting program on the shelf" activity -- creating programs that nobody will use.  My only hope is that  "back to basics" movement will strengthen with time.

The trend now is toward "integrated" configuration management solutions like Ansible ( reinvention of IBM JCL on a new level), which have their own weak spots  And it is difficult to fight against fashion in software, much like in women cloth.  But again, some of the utilities listed below can be used as steps in Ansible playbooks you develop.

Does continuing this project make any sense?

This is an open question. Of couse illustration of defensive programming principles (as imperfect as they are in this particular case) on small sysadmin utilities has some educational value beyond  and above functionality provided.  But when enough is enough ? Finishing more of my existing utilities to the level acceptable for publication is a very labor intensive process. and probably makes little sense as the key principles and architecture used are already well illustrated on existing set. I recently spend 12 hours on "finishing touches" for adding to the collection a simple utility as dormant_user_stats ;-).  Which is a tiny and a very basic utility, essentially a more intelligent, more polished Perl implementation of a three line Bash script: 

find /home -maxdepth 1 -type -d  -mtime +365 | xargs -L 1  du -sh
du -dh /home
find /home | wc -l

Instead of basic output   dormant_user_stats  produces a neat table with all kind of additional stats required for deciding whether reclaiming the space is worth the candles burned in the process, and if yes, for which 20% of users in the list above it should be done.  It does one thing, important for system administration of HPC clusters, large Web sites,  and similar environments where large number of directories are present (home directories or branches of the WEB site, etc). But polishing it does not justify the effort as my private version has the same functionality while being probably 50% shorter in LOC metric.

Estimating the number of  "dead" branches for maintainers of web sites can be easily done with other tools. and cases where, say we have , say, over a hundred of hone directories on a server are no longer common. Only in those cases knowing what branches/home directories are stale and need to be iether updated or archived, saving space and i-nodes, is as important

When the number of users often exceeds 300 primitive bash script like above can run several hours and you can benefit form some optimization which  dormant_user_stats implements (which first of all means more careful filtering). But in any case such utilities can run several hours (on petabyte storage),  especially if multiple users have multi-terabyte directories and millions of files.

It might ne simple just to compress the "suspect" directories and  see if the user complain ;-) I have another utility called dir2tar which does this safely with all kind of verifications in the process and can be used along with  dormant_user_stats for the top 20% of dormant users or, more correctly, dormant directories.

The list of utilities I eventually plan to publish

Among around 200 utilities which I have written during my carrier, the following might have some value to other sysadmin:

1. [Published] Defensive diagnostic framework module. It provides  the following functionality:

   1. All messages are produced with the line in which they occurred.
   2. Messages are generated using subroutine logme which has two parameters (error code and the text of the message). They are printed with the error code, which signify the severity and the line number at which logme subroutine was involved. Four levels are distinguished:
      • Warnings: informational messages that do not affect the validly of the program output or any results of its execution. Still the situation that deserve some attention
      • Errors: (correctable errors) Messages that something went wrong but the results execution of the program is still OK and/or output of the program most probably is still valid
      • Severe errors (failures). Program can continue but the results are most probably a garbage and should be discarded. Diagnostic messages provides after this point might still have a value.
      • Terminal or internal errors (abends). Program can't continue at this point and need to exit. In case of cron jobs, for such abnormal situations you can even try to email the developer.

      To achieve this one needs to write or borrow and adapt a special messages generation subroutine for example logmes, modeled after one used in compilers. One of the parameters passes to this subroutines should be the one byte code of the error (or its number equivalent) along with the line in which error was detected.  For example

      • W -- warning --  the program can continue; most probably results will be correct, but in rare cases this situation can lead to incorrct or incomplete results.
      • E -- error  -- continuation of the program possible and the result most probably will be correct (correctable errors)
      • S -- failure (serious error)  -- continuation of program possible but the result most probably will be useless or wrong.
      • T -- internal error 

      The abbreviated string for those codes has the mnemonic string WEST

   3. Terminal errors (abends) represent a special case, for which some additional diagnostic should be provided. They are invoked with a special subroutine (called abend), which can provide extended context for the error
   4. Output of those message is regulated by option verbosity (-v). The highest level is 3 which produces all errors. 
   5. There are also two types of informational messages that can be produced  (which are not suppressible):
      1. "I" (informational) messages produced with the error code. They can be generated using logme subroutine
      2. Regular output which is produced without error code, as is. They can be generated using out subroutine
   6. There is  a possibility to output of the summary of the messages at the end of execution of programs
    
2. [Published] neatperl -- simple and pretty safe prettyprinter for Perl. See Neatperl -- a simple Perl prettyprinter. Generated listing, which  contains a column with nesting (like in old IBM PL/1 compliers) , which is a great helps for detection of unclosed curvy parenthesizes errors in complex deeply nested scripts. Beatification affects only leading space of each line which minimize chances of corruption of the script by the beautifier, which is a real danger for a language with such complex lexical structure as Perl. Also produces heuristically complied and only partially correct, but still useful XREF table, which allows you see the general map of identifiers usage in the program (the capability that Perl interpreter, sadly, is missing).
    
3. [Published] neatbash -- simple, limited but safe prettyprinter for bash.  Beautifier for bash. Like Neatbash  beatification affects only leasing space of each line. Does some sanity check and helps to detect unclose control statement delimiter errors.
    
4. [Published] saferm -- Wrapper for rm that helps to prevent accidental deletion of vital files  See Creative uses of rm
    
5. [Published] dormant_user_stats -- lists users who were inactive for the specified number of days (default is 365). Calculates I-nodes usage too.  Can execute  simple commands for each dormant user (lock or delete account) and generates a text file with the list (one user per line) that can be used for more complex operations.
    
6. [Published] emergency_shutdown -- Useful for large RAID5 arrays without spare drive, or other RAID configurations with limited redundancy and critical data stored. Currently works with Dell DRAC only, which should be configured for passwordless ssh login from the server that runs this utility. Detects that a disk in RAID 5 array failed, informs the most recent users (default is those who login during the two months), and then shuts down the server, if not cancelled after the "waiting" period (default is five days).
    

7. [Published] msync --  rsync wrapper that allow using multiple connections for transferring compressed archives or sets of them orginized in the tree (which are assumed iether consist of a single files to a subdirectory with the archive split into chunks of a certain size, for example 5TB ) . Files are sorted into N piles, where N is specified as parameter,  and each pile transmitted vi own TCP connection. useful for transmitted over WAN lines with high latency. I achieved on WAN links with 100Ms latency results comparable with Aspera using 8 channels of transmission.

   It organizes files into given number of "piles" and transfers all piles in parallel using for each separate invocation of rsync or tar pipe to the target server. If program is terminated before full transfer took place, on next execution partial transfers are completed using rsync. Each time it will start from the point it was terminated operating like rsync but for the whole directory tree or file list. It scans the content of the remote size and transmit only those files what not yet transmitted.

   The script tried to verify whether files were already transferred and if missing files are detected it transfers only missing files (partially transferred files transferred using rsync)
    

8. [Published] usersync -- allows to manage small set of users without any directory or NIS. It synchronizes (one way) users and groups within given interval of UID (min and max)  with the directory or selected (etalon) remote server (files should be accessible via ssh.) Useful for provisioning multiple servers that use traditional authentication, and for synchronizing user accounts between multiple versions on Linux without LDAP or Active Directory.  Also can be used for "normalizing" servers after acquisition of another company, changing on the fly UID and GID on multiple servers, etc.  Can also be used for provisioning computational nodes on small and medium HPC clusters that use traditional authentication without NIS. 
    
9. [Published]  dirhist -- History of changes in one or several directories. Designed to run from cron. Uses different, simpler approach than the etckeeper. Does not use GIT or any other version control system as they proved to be of limited use for sysadmin, unless there are multiple sysadmins on the server. Creates a daily new tar file for selected set of directories, for example  /etc, /root, and /boot  Can be integrated with GIT for pushing all changes into GIT automatically

   Detects all "critical" changed file, diffs them with previous version, and produces report. All information by default in stored  /var/Dirhist_base (configurable via  via config file.) You can specify "watched" files and directories within those folders in a format similar to YAML dictionaries, or Windows 3 ini files. 

   If any of "watched" files or directories changes the utility can email you the report to selected email addresses, to alert about those changes. Useful when several sysadmin manage the same server.  Can also be used for checking, if changes made were documented in GIT or other version management system with the utility admpolice.
    

10. [Published] fspace_mon --  Intelligent filesystems free space monitor.  "Primitive", one level free space monitoring program can be written in an hour or so, but intelligent monitoring of free space with the suppression of redundant messages a nd flexibility of to who for particular filesystem you need to send emails and what action you need to take is the last critical threshold is breached, is difficult without using some kind of correlation engine. This utility tries to imitate correlation engine and provides three level of alerts (Warning, Serious, and Critical/Fatal) for free space with the ability to specify a mail list (individual for each monitored filesystem) and an action for the last level (also for each monitored filesystem; action can be a shutdown of the server, cleaning some folder,  or blocking of user logins). Emails warning about insufficient disk space sent to the most recent users, or specified mail list.  Blocks "spam" emails and sends exactly one email after crossing each threshold. Allow to specify fractions of percentage for the last (critical) threshold, for example 99.99%.
     
11. [Published] think -- "Think it over first" (TIOT) type of utility

    Very useful for working with remote or critical production server, where a mistake or accidental typo can cost a lot pain/money. Originally was used for preventing accidental reboots, but later made more universal. It allows to create a set of aliases, sourcing which will prevent execution of the command if it is submitted in interactive session printing inread the context in which particular command will be executed (customizable via configuration file.) Operated via concept of dangerous options, set of which can be specified for each command to alert sysadmin about possible tragic consequences' or a rush. impulsive  run of such a command, typical when sysadmin is stressed and/or is working under tremendous pressure.  In such circumstances it is important not to make the situation worse. So if you type the command reboot the utility will print the HOSTNAME of the server you are on,  and will ask you to resubmit the command.  while for a command like find  it provides PWD  the list of "dangerous options" used, if any.  Useful when working on remote server, which in case you do something nasty might require you to buy an airline ticket and fly to the destination instead of TCP packets.
     

12. [Published] soft_reboot  -- reboots the server only after a given period since the last reboot( by default 90 days) expires, selecting a moment when the server does not run any jobs, as shown by uptime.  Useful for computational cluster nodes, especially if some applications suffer from memory leaks.
     
13. [Published] dir_favorites.pl  Help to create a usable history of directory favorites. It requires the usage of custom prompt function (provided in  dir_favorites_shell_functions.sh.)  Favorites are stored in stored in stack accessible by dirs command and consist of two parts -- static (loaded from a file  $HOME/.config/dir_favorites.$HOSTNAME ) and dynamic (created from history).   The utility creates a history of accessed directories and a set of aliases such as cd--, 3rb for root/bin,  2esn for /etc/sysconfig/network-scripts. You got the idea.

    It requires the usage of custom prompt function (provided as my_prompt function in the distribution file dir_favorites_shell_functions.sh)  which should be referenced in .bash_profile/.bashrc as:

    export PROMPT_COMMAND='my_prompt'

    The second function that uses this utility is function go ( also provided in dir_favorites_shell_functions.sh). It displayed list of directories  and allows to cd to any of them by putting the number of copying and editing the path. For example:

     [0]root@lustwz99: # go
     0 ~/.config
     1 ~/bin
     2 ~/.config
     3 /etc/sysconfig
     4 /etc/sysconfig/network-scripts
     5 /etc/profile.d
     6 /var/log
     7 /etc --- the divider between static part and dynamic part of favorites; directors below are generated dynamically from the history
     8 /tmp/rhel
     9 /tmp/rhel/isolinux
     10 /mnt/Packages
     ... ... ... ..
     Select directory by the number (negative from bottom) or copy and edit the path:

    Favorites are stored in Bash directory stack accessible by dirs command. and so on as well as via function go also provided in dir_favorites_shell_functions.sh

    If it is invoked with option -m also  changes directory favorites in MC (Midnight commander)  providing dynamic updates. In this case mc hotlist consist of two parts -- static and dynamic.
     

14. [Published] dir2tar -- compresses the directory replacing all files in it with the archive and creating manifest (essentially the list of tarball content). Uses pbzip2 for compression by default. Useful if resulting archive is over 100GB as it checks for interruptions in archiving and many other abnormal situations. It also recognized already archived directories. Can work in tandem with the  dormant_user_stats. Compression program used can be changed via option -a, for example -a pigz . For obvious reasons for large archives only parallel versions of compression programs are acceptable. Tested on tarballs up to 20TB.
     
15. [Published] centos2ol_wrapper  Wrapper with sanity checks for centos2ol. It incorporates several "no nonsense" checks that make success of the conversion more probable. I experienced around 30% failure rate iin my tests and 10% (or one server out of ten failed to report after the conversion was finished). Serious troubles include but not limited to  deletion of hundreds of vital RPMs ( due to my mistake made out  of frustration; nit stll no protection from this kind  of errors), if safety measures are ignored. So failures due to the lack of pre-conversions checks is not a hypothetic scenario, especially failure on the state of rolling our Oracle RPMs, when the system in "transitional state" and can't be safely rebooted. It is an important safety measure if you convert multiple CentOS or RHEL servers to Oracle Linux and need to convert  important production servers. See  Converting CentOS to Oracle Linux with centos2ol script for details
     
16. [Published] eg_install.sh Installation script for eg which allow to reuse tldr pages and edit them creating private knowledgebase with examples of major Linux utilities.

    It is impossible to remember details of Linux utilities: there are too many of them and some are quite complex (find, xargs, grep, systemctl, etc). To compensate for the low quality on Linux man pages (which typically lack meaningful examples) you can use community database (for example tldr database available from GitHub; it can be viewed using Web browser from ostera.io) or to create a private database of examples, partially or fully extracted from .bash_history. The latter allows to adapt examples to your needs and as such is closer to the idea of a knowledgebase, although one does not exclude another.

    I experimented with adaptation for this purpose of Python script eg, available from GitHub and got encouraging results. The idea is to supplement the default set of pages with examples provided with eg with pages from tldr and gradually enhance and edit them by incorporating own examples from .bash history.

    The script eg uses two directories for storing the examples files (one for each utility; each written in the subset of Markdown)

    1. custom directory -- the directory in which you can edit files using option -e (for example eg -e find)
    2. default directory -- the directory for the small set of examples which comes with the script.

    If pages for the particular tool/utility are present in both, eg merges them creating composite page. So it is possible for example to store your own example in custom directory, and tldr in default directory but I prefer to edit tldr pages directly.

    To create a private knowledgebase of examples that uses the suggested above format with set of tldr pages as the initial content of the knowledgebase, you need first to download two zip files from GitHub eg-master.zip and tldr-main.zip to your home directory. That allows to avoid pains with proxy.

    After that run the following installation script. Alias eg needs to be added to your .bash_profile or to .bashrc, depending on how your dot files are organized. The script adds it .bash_profile which might be incorrect.

    See Admin/Sp_admin_utils/eg_install.shtml and Linux command line helpers for more details
     

17. intelliping -- ping all servers from the list containing both DRAC/ILO addresses and main interface IP (pinging separately DRAC/ILO and main interface) and informs if the power was restored, or the server rebooted. Reports situations when DRAC/ILO interface is on, but the main interface is not available. Or vise versa.  Uses nmap to confirm the diagnosis and compare the current state with the default map of open ports for each server. 
     
18. inform_recent_users -- allow sending message only to users who login to the server for the given period (default 90 days). Set of such users is determined via last command. Useful for a large server park shutdown.
     
19. selectivezip -- by default compress using pbzip2 files with specified extensions, and/or files with sizes above given threshold. Set of extensions to use should be provided via config file.  Useful for compression FASTA files in large genomic trees.
     
20. reboot_sanity_chk -- Designed to run from cron on reboot. The script that checks if all necessary filesystem were mounted, interfaces are up after the reboot, and if not report the problems to sysadmin and attempt to mount missing NFS shares again. It also performs several other sanity checks related to RHEL/CentOS servers.
     
21. sign -- small utility that signs GIT change during commit -- adds name of sysadmin to the Git comment for the change. Important when sysadmin is working as root on the system that has multiple sysadmins. That provides track record of each sysadmin actions, so that they are better aware of changes make by all others.  
     
22. heartbeatmon -- simple monitoring multiple servers via heartbeat on common NFS or GPFS filesystem. Can be used with Nagios.
     
23. mkuser -- create the same user on multiple servers using pdsh, cexec or ansible. Creates .ssh directory and configures passwordless login. Written for use in small HPC clusters. Can recompile a record from /etc/passwd into set of groupadd and useradd commands.
     
24. useractivity -- collects statistics including IPs of user logins from /var/log/wtmp and tries to detect the user pattern of usage login to the server. If runs on multiple servers the  information will be merged. Produces a monthly or bi-montly or quarterly table of access of each user to the large server farm.
     
25. admpolice -- verifies that all changes in /etc or other selected directory are all documented in GIT or version control system. It verifies all changes detected via dirhist were documented in GIT (etckeeper can also be used),  and that comment for the change contains the initials of sysadmin, who made this change. Useful if multiple sysadmin exists for the server.
     
26. sysinfo -- a simple baseliner and collector of critical information needed for the recovery in case the server crashed in such way that OS partition is destroyed (happens with RAID5 arrays if two disks are failed). Generates report in  HTML. It uses a different, much simpler approach then sosreport or other baseline utilities.

    Written long ago, during the period when a 4GB USB drive was large. Used mainly for creating a set of HTML pages describing managed servers in some details as a party of creation of inventory.  Allows custom fields for putting information not collectable automatically.  Can be used via Ansible.

    In view of existence of Relax-and-Recover  and cheap 256GB Fit drives might not be that important to have, although it collects some dynamic configuration information from the current session (output of ifconfig and other utilities that provide the current map of connections, disk map, etc.) Useful on small cluster, especially if you use SGI as your scheduler.

NEWS CONTENTS

• 20210604 : Softpanorama defensive diagnostic framework module ( Softpanorama defensive diagnostic framework module, Jun 04, 2021 )
• 20210505 : eg_install.sh Installation script for eg which allow to reuse tldr pages and edit them creating private knowledgebase with examples of major Linux utilities ( softpanorama.org, May 05, 2021 )
• 20201228 : centos2ol_wrapper ( centos2ol_wrapper, Dec 28, 2020 )
• 20201221 : dir2tar ( dir2tar, Dec 21, 2020 )
• 20201123 : fspace_mon -- Intelligent filesystems free space monitor ( fspace_mon -- Intelligent filesystems free space monitor, Nov 23, 2020 )
• 20201123 : think -- "Think it over first" (TIOT) type of utility ( think -- "Think it over first" (TIOT) type of utility, Nov 23, 2020 )
• 20201123 : soft_reboot -- reboots the server only after a given period since the last reboot ( by default 90 days) selecting a moment when the server does not run any jobs, as shown by zero uptime ( soft_reboot -- reboots the server only after a given period since the last reboot ( by default 90 days) selecting a moment when the server does not run any jobs, as shown by zero uptime, Nov 23, 2020 )
• 20201123 : dir_favorites.pl Create a dynamic history of directories, assesible via provided function go (or via dirs command) Can use static favorites list too ( dir_favorites.pl Create a dynamic history of directories, assesible via provided function go (or via dirs command) Can use static favorites list too, Nov 23, 2020 )
• 20201104 : Utility dirhist -- History of changes in one or several directories ( Utility dirhist -- History of changes in one or several directories , Nov 04, 2020 )
• 20201102 : Utility usersync which synchronizes (one way) users and groups within given interval of UID (min and max) with the directory or selected remote server ( Utility usersync which synchronizes (one way) users and groups within given interval of UID (min and max) with the directory or selected remote server , Nov 02, 2020 )
• 20201030 : Utility msync -- rsync wrapper that allow using multiple connections for transferring compressed archives or sets of them orginized in the tree ( softpanorama.org, Oct 30, 2020 )
• 20201027 : Utility emergency_shutdown ( Utility emergency_shutdown , Oct 27, 2020 )
• 20201019 : The utility dormant_user_stats ( The utility dormant_user_stats, Oct 19, 2020 )

Old News ;-)

[Jun 04, 2021] Softpanorama defensive diagnostic framework module

A short description can be found at Softpanorama defensive error diagnostic framework

It provides the following functionality:

1. All messages are produced with the line in which they occurred.
2. Messages are generated using subroutine logme which has two parameters (error code and the text of the message). They are printed with the error code, which signify the severity and the line number at which logme subroutine was involved. Four levels are distinguished:
   • Warnings: informational messages that do not affect the validly of the program output or any results of its execution. Still the situation that deserve some attention
   • Errors: (correctable errors) Messages that something went wrong but the results execution of the program is still OK and/or output of the program most probably is still valid
   • Severe errors (failures). Program can continue but the results are most probably a garbage and should be discarded. Diagnostic messages provides after this point might still have a value.
   • Terminal or internal errors (abends). Program can't continue at this point and need to exit. In case of cron jobs, for such abnormal situations you can even try to email the developer.

   To achieve this one needs to write or borrow and adapt a special messages generation subroutine for example logmes, modeled after one used in compilers. One of the parameters passes to this subroutines should be the one byte code of the error (or its number equivalent) along with the line in which error was detected. For example

   • W -- warning -- the program can continue; most probably results will be correct, but in rare cases this situation can lead to incorrct or incomplete results.
   • E -- error -- continuation of the program possible and the result most probably will be correct (correctable errors)
   • S -- failure (serious error) -- continuation of program possible but the result most probably will be useless or wrong.
   • T -- internal error

   The abbreviated string for those codes has the mnemonic string WEST

3. Terminal errors (abends) represent a special case, for which some additional diagnostic should be provided. They are invoked with a special subroutine (called abend), which can provide extended context for the error
4. Output of those message is regulated by option verbosity (-v). The highest level is 3 which produces all errors.
5. There are also two types of informational messages that can be produced (which are not suppressible):
   • "I" (informational) messages produced with the error code. They can be generated using logme subroutine
   • Regular output which is produced without error code, as is. They can be generated using out subroutine
6. There is a possibility to output of the summary of the messages at the end of execution of programs

[May 05, 2021] eg_install.sh Installation script for eg which allow to reuse tldr pages and edit them creating private knowledgebase with examples of major Linux utilities

It is impossible to remember details of Linux utilities: there are too many of them and some are quite complex (find, xargs, grep, systemctl, etc). To compensate for the low quality on Linux man pages (which typically lack meaningful examples) you can use community database (for example tldr database available from GitHub; it can be viewed using Web browser from ostera.io) or to create a private database of examples, partially or fully extracted from .bash_history. The latter allows to adapt examples to your needs and as such is closer to the idea of a knowledgebase, although one does not exclude another.

I experimented with adaptation for this purpose of Python script eg, available from GitHub and got encouraging results. The idea is to supplement the default set of pages with examples provided with eg with pages from tldr and gradually enhance and edit them by incorporating own examples from .bash history.

The script eg uses two directories for storing the examples files (one for each utility; each written in the subset of Markdown)

1. custom directory -- the directory in which you can edit files using option -e (for example eg -e find)
2. default directory -- the directory for the small set of examples which comes with the script.

If pages for the particular tool/utility are present in both, eg merges them creating composite page. So it is possible for example to store your own example in custom directory, and tldr in default directory but I prefer to edit tldr pages directly.

To create a private knowledgebase of examples that uses the suggested above format with set of tldr pages as the initial content of the knowledgebase, you need first to download two zip files from GitHub eg-master.zip and tldr-main.zip to your home directory. That allows to avoid pains with proxy.

After that run the following installation script. Alias eg needs to be added to your .bash_profile or to .bashrc, depending on how your dot files are organized. The script adds it .bash_profile which might be incorrect.

See Admin/Sp_admin_utils/eg_install.shtml and Linux command line helpers for more details

[Dec 28, 2020] centos2ol_wrapper Wrapper with sanity checks for centos2ol.

It incorporates several "no nonsense" checks that make success of the conversion more probable. I experienced around 30% failure rate iin my tests and 10% (or one server out of ten failed to report after the conversion was finished). Serious troubles include but not limited to deletion of hundreds of vital RPMs ( due to my mistake made out of frustration; nit stll no protection from this kind of errors), if safety measures are ignored. So failures due to the lack of pre-conversions checks is not a hypothetic scenario, especially failure on the state of rolling our Oracle RPMs, when the system in "transitional state" and can't be safely rebooted. It is an important safety measure if you convert multiple CentOS or RHEL servers to Oracle Linux and need to convert important production servers. Man page at centos2ol_wrapper. See Converting CentOS to Oracle Linux with centos2ol script for details

[Dec 21, 2020] dir2tar -- compresses the directory replacing all files in it with the archive and creating manifest

Uses pbzip2 for compression by default. Useful if resulting archive is over 100GB as it checks for interruptions in archiving and many other abnormal situations. It also recognized already archived directories. Can work in tandem with the dormant_user_stats. Compression program used can be changed via option -a, for example -a pigz . For obvious reasons for large archives only parallel versions of compression programs are acceptable. Tested on tarballs up to 20TB.

[Nov 23, 2020] fspace_mon -- Intelligent filesystems free space monitor

"Primitive", one level free space monitoring program can be written in an hour or so, but intelligent monitoring of free space with the suppression of redundant messages and flexibility as for whom for particular filesystem you need to send emails and what action you need to take is the last critical threshold is breached, is difficult without using some kind of correlation engine.

This utility tries to imitate correlation engine and provides three level of alerts (Warning, Serious, and Critical/Fatal) for free space with the ability to specify a mail list (individual for each monitored filesystem) and an action for the last level (also for each monitored filesystem; action can be arbitrary command or script, for example a shutdown of the server, cleaning some folder, or blocking of user logins).

Emails warning about insufficient disk space sent to the most recent users, or specified mail list. Blocks "spam" emails and sends exactly one email after crossing each threshold. Allow to specify fractions of percentage for the last (critical) threshold, for example 99.99%.

[Nov 23, 2020] think -- "Think it over first" (TIOT) type of utility

Very useful for working with remote or critical production server, where a mistake or accidental typo can cost a lot pain/money. Originally was used for preventing accidental reboots, but later made more universal. It creates a set of aliases for several "potencially dangerious" utilities(rm, find, chmod, chown, etc), sourcing which will prevent execution of the command if it is submitted in interactive session printing instead the context in which particular command will be executed (customizable via configuration file.)

Operated via concept of dangerous options, set of which can be specified for each command in the config file for this utility. If and of "dangerous options" is detected, the utility will alert sysadmin about possible "tragic consequences" of rush, impulsive run of such a command, typical when a sysadmin is stressed, tired/sleepy/overcafienated, and/or is working under tremendous pressure. Extreme fatigue and exhaustion often leads to SNAFU that makes an existing difficult situation far worse.

Or just distracted or absent minded who failed to concentrate and access the environment in a proper way.

If option LS is specified in config file for this utility and dangerous options are detected the command tried to convent the command into an ls command and execute it to give you better understanding about which file are affected. Sysadmins know that this is one of the best way to prevent SNAFU with find -exec (or find -delete, or chmod -LR something) but seldom follow this recommendation.

In such circumstances it is important not to make the situation worse. So if you type the command reboot the utility will print the HOSTNAME of the server you are on, and will ask you to resubmit the command. while for a command like find it provides PWD the list of "dangerous options" used, if any. Useful when working on remote server, which in case you do something nasty might require you to buy an airline ticket and fly to the destination instead of TCP packets.

Again, one of the most useful features of this command is option LS, which you can specify in config file for any command, and which attempts to change the command such a find with exec option and executing the same command with '-ls' instead. For command like rm, chmod and chown this is simpler: you just need discard all options and provide ls with "naked arguments". But of course the devil is in details as some arguments might contain spaces, or special characters (like -name in find or the first argument in grep) and need to be quoted.

Delay serves as a safety valve that allow you to correct some dangerous typos, automatically specifying absolute path for the backup of system directory (/etc instead of etc), and like.

[Nov 23, 2020] soft_reboot -- reboots the server only after a given period since the last reboot ( by default 90 days) selecting a moment when the server does not run any jobs, as shown by zero uptime

Useful for computational cluster nodes, especially if some applications suffer from memory leaks.

[Nov 23, 2020] dir_favorites.pl Create a dynamic history of directories, assesible via provided function go (or via dirs command) Can use static favorites list too

Help to create a usable history of directories most frequently accessed (favorites). It requires the usage of custom prompt function (provided in dir_favorites_shell_functions.sh.) Favorites are stored in stored in stack accessible by dirs command and consist of two parts -- static (loaded from a file $HOME/.config/dir_favorites.$HOSTNAME ) and dynamic (created from history). The utility also dynamically generates aliases such as cd--, cd---, cd-4, cd-5 and so on as well as via function go also provided in dir_favorites_shell_functions.sh

If it is invoked with option -m also changes directory favorites in MC (Midnight commander) providing dynamic updates. In this case mc hotlist consist of two parts -- static and dynamic.

NOTE: the set of shell function which are necessary for this ustility is provided in dir_favorites_shell_functions.sh

[Nov 04, 2020] Utility dirhist -- History of changes in one or several directories

Designed to run mainly from cron. Uses a different, a much simpler approach to the same problem than the etckeeper and extends it to arbitrary number of directires. Does not use GIT or any other version control system as they proved to be of questionable utility for syadmins, unless there are multiple sysadmins on the server.

It does not have the connected with the usage of GIT problem with incorrect assignment of file attributes when reconverting system files.

If it detects changed file it creates a new tar file for each analyzed directory. For example /etc, /root, and /boot

Detects all "critical" changed file, diffs them with previous version, and produces report.

All information by default in stored in /var/Dirhist_base. Directories to watch and files that are considered important are configurable via two config files dirhist_ignore.lst and dirhist_watch.lst which by default are located at the root of the /var/Dirhist_base tree ( as /var/Dirhist_base/dirhist_ignore.lst and /var/Dirhist_base/dirhist_watch.lst )

You can specify any number of watched directories and within each directory any number of watched files and subdirectories. The format used is similar to YAML dictionaries, or Windows 3 ini files. If any of "watched" files or directories changes, the utility can email you the report to selected email addresses, to alert about those changes. Useful when several sysadmin manage the same server. Can also be used for checking, if changes made were documented in GIT or other version management system (this process can be automated using the utility admpolice.)

For more information see dirhist utility

[Nov 02, 2020] Utility usersync which synchronizes (one way) users and groups within given interval of UID (min and max) with the directory or selected remote server

Useful for provisioning multiple servers that use traditional authentication, and not LDAP and for synchronizing user accounts between multiple versions on Linux . Also can be used for "normalizing" servers after acquisition of another company, changing on the fly UID and GID on multiple servers, etc. Can also be used for provisioning computational nodes on small and medium HPC clusters that use traditional authentication instead of DHCP.

[Oct 30, 2020] Utility msync -- rsync wrapper that allow using multiple connections for transferring compressed archives or sets of them orginized in the tree

Useful for transfer of sets of trees with huge files over WAN links. In case of huge archives they can be split into chunks of a certain size, for example 5TB and orgnized into a directory. Files are sorted into N piles, where N is specified as parameter, and each pile transmitted vi own TCP connection. useful for transmitted over WAN lines with high latency. I achieved on WAN links with 100Ms latency results comparable with Aspera using 8 channels of transmission.

[Oct 27, 2020] Utility emergency_shutdown

Useful for large RAID5 arrays without spare drive, or other RAID configurations with limited redundancy and critical data stored. Currently works with Dell DRAC only, which should be configured for passwordless ssh login from the server that runs this utility. Detects that a disk in RAID 5 array failed, informs the most recent users (default is those who login during the two months), and then shuts down the server, if not cancelled after the "waiting" period (default is five days).

[Oct 19, 2020] The utility dormant_user_stats

The utility lists all users who were inactive for the specified number of days (default is 365). Calculates I-nodes usage too. Can execute simple commands for each dormant user (lock or delete account) and generates a text file with the list (one user per line) that can be used for more complex operations.

Use

dormant_user_stats -h

for more information

Recommended Links

Softpanorama Recommended

Top articles

Sites

• Sysadmin Horror Stories
• Enterprise Unix Administration
• Parallel command execution
• Unix System Monitoring
• Unix Configuration Management Tools

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: June 04, 2021