Nikolai Bezroukov. Portraits of Open Source Pioneers

For readers with high sensitivity to grammar errors access to this page is not recommended :-)

Bourne Shells Family

Copyright 1996-2005, Dr. Nikolai Bezroukov. This is a copyrighted unpublished work. All rights reserved.

"The lyfe so short, the craft so long to lerne,''
Chaucer c. 1340–1400
(borrowed from the bash FAQ)

Few people understand that classic Bourne-style shells are really powerful (albeit slightly outdated) scripting language. And despite their age in some respects they are still unsurpassed as very high level languages.  It is actually shells that make the power of Unix kernel available to ordinary people :-).

Korn shell was the oldest of a series of  successful scripting languages produced by Unix  (ksh was probably the first true "glue" language, awk added some new data processing capabilities that were later integrated by Perl into the Bourne-Cshell-Kornshell framework). Because it was invented for a very flexible Unix environment, Unix shells incorporate some non-trivial capabilities and are underutilized by most users.  The fact that Bourne/Korn shell survived for so many years despite the availability of alternatives suggests that it probably represented the first "good enough" implementation of scripting language.

Moreover "Shell+C" combination was the first practical components-based programming  methodology: an important idea here is to use rather small C programs and preexisting Unix tools as a sort of Lego to create a solution of the problem with the shell serving as a glue. That approach can be very successful even today (in the days of monstrous middleware and Java misunderprogramming ;-) and  it is definitely remains one of the best approaches in creating prototypes for complex systems.  In a sense Unix promotes LEGO approach to building prototypes and a special ("unixoidal") type of thinking about architecture in terms available in Unix (including such powerful protocols as pipes and sockets).  In case this way of thinking and componentization compete with architectural thinking of talented people who did not have exposure to Unix the difference in favor of Unix style thinking can be pretty dramatic.  This was actually the case even when on the opposite side was Donald Knuth.

It's very important to realize that shell in general and Korn shell in particular to a certain extent is a quintessence of Unix programming methodology. As such Korn shell is pretty close to the crown jewel of Unix as a programming environment and it does deserve in-depth study.

I would like to stress again that both Bourne-shell and Korn-shell were not "instant" final products. They were incremental improvements in a long chain of development (Bourne shell is based on Mashey shell, which in turn was an extension of the original Thompson shell, which in turn was a derivative of Multics shell ;-). 

As John Mashey, the author of Mashey shell (the precursor of the Bourne shell and the successor of the original Thompson shell, see History), mentioned (personal correspondence), people in Bell Labs instantly realized that shell represents a new a highly productive programming environment: 

People all over BTL wrote shell scripts, but anything big was just too painful in the original [Thompson] shell, so we added simple variables, reasonable control structures, the variable path search which meant that individuals and groups could establish their own libraries of scripts not in /bin or /usr/bin], and pre-initialized variables for home directory. As a result, in BTL Piscataway, there was an explosion in the use of shell as a scripting language, and there were large number of stored scripts. There was some argument with Steve later, as PY had accumulated so much of this, that the slow speed of the original Bourne shell implementation would have been very painful.

I would like to stress that at this time Bell Labs were populated by very talented people who became pioneers in several areas of OS design. Still the invention of scripting is one of their top achievements. Among other achievements that directly influenced shell programming and shell language capabilities were pipes, regular expressions and the hierarchical filesystem. BTW the level of integration of those facilities into shell (especially named pipes and regular expressions) are still unsurpassed (Perl has a better regular expressions integration, but its pipes support simply sucks). Unfortunately those capabilities are under-appreciated and under-used by most users and system administrators.  This page is the result of my desire somewhat improve this situation ;-).

It truly unfortunate that many capabilities of shell that substantially increase its usefulness are a kind of esoteric art. Even good .profile and .kshrc files are a such a rare thing that can be used as a good interview question for advanced Unix sysadmin positions ;-(. As David Korn aptly said (bold italic is mine  - BNN):

There are many people who use UNIX or Linux who IMHO do not understand UNIX. UNIX is not just an operating system, it is a way of doing things, and the shell plays a key role by providing the glue that makes it work. The UNIX methodology relies heavily on reuse of a set of tools rather than on building monolithic applications. Even perl programmers often miss the point, writing the heart and soul of the application as perl script without making use of the UNIX toolkit.

It's probably true that Perl is better for any sizable programming project. And IMHO it is one of the few scripting languages that to a large extent retained the spirit of Born-Korn-Cshell troika. But as I already mentioned above,  the problem is that Perl pipe support is weak, and that makes many tasks that involve their usage inconvenient.  Actually, this shortcoming makes Perl much less expressive then ksh93 and bash for the large class that can be solved using pipes.  Due to this shortcoming often Perl programmers are implicitly forced to reinvent the bicycle (often in a form of loops) instead of writing a pipe. But other than that, Perl it's a better, higher level language with much better debugging capabilities and with a good integration with Unix API.

Still you can and should use shell for programming in cases where simplicity and transparency is very important. For programming  among a half dozen of available shells I recommend ksh93. Bash is a better interactive shell, but  not for programming, especially outside Linux environment.  It also suffers from feature creep. While ksh93 has some interesting programming constructs and is more clean design (and especially code) then bash, no claim of exclusivity is made, especially as shells copy one another.

Bash 3.x  are now de-facto standard among open source shells. Actually bash was the best free shell until ksh93 became freely available. Now situation is not that clear and ksh93 has a definite edge if programmability or quality of implementation are important. Bash still holds his own as an interactive shell (Bash  3.x has a built-in interactive debugger that is more than a diagnostic tool,  it is a fine way to explore the very guts of bash), but you should consider using Orthodox File Manager like mc to use bash fully.

Generally, I would say that FSF dropped the ball and the product development was stalled.  The implementation became too baroque. The quality and elegance of ksh93 implementation is higher and its port to windows is very impressive (the best port of Unix shell ever).  The only important enhancement for the last decade was the built-in debugger added in version 3.0.  The latter is a very nice addition and I never understood why nobody added a simple debugger to the shell after, say, 1990, when PC became too powerful for personal computers anyway :-).

What is really important is that ksh93 has superior pipes support  both in comparison with other shells and Perl. From the programming language design point of view ksh93 has better integrated with piping constructs that other widespread shells (rc shell is the only more powerful in this respect).  BTW very few people know and understand that dtksh, the Desktop KornShell that comes standard with CDE on all commercial Unixes is an extension of ksh93 that contains a set of built-in commands to interface with Motif, but can be used as a regular shell as well. For example in Solaris 9 it can be found at /usr/dt/bin/dtksh.

The only other shell that has advanced piping facilities is rc-shell and that actually makes rc-shell sometimes superior to Perl in scripts that use heavy piping. See also Softpanorama Perl bibliography.

Here we will discuss using shell in Unix environment. But it can be used both in Windows environment too, although in the Windows environment it's much less powerful and stable. But my students did some amazing things with Uwin  -- ksh93 in Windows environment (preferable to Cygwin from the quality standpoint, but less popular).  BTW for NT and Win2K you can also try Jscript -- a very interesting and a pretty novel shell that shares with REXX a very important capability to be used both as shell and macrolanguage (the most attractive feature of OS/2 and Amiga).  Jscript which is a pretty conformant implementation of JavaScript that provides an unique advantage of using the same language for HTML scripting and shell programming.  For Ms Office JScript can be used instead of VBA. I do not recommend VBScript, but your mileage may vary, as most examples and books are devoted to this language.

Other than more powerful piping constructs, ksh93 has some other nice features:

• It has so called "discipline functions": essentially exceptions called on access to a particular shell variable.

• It  supports FPATH variable (one of the missing features in bash and pdksh; very useful for putting tested/debugged functions into files that the FPATH points to).

• ksh93 is noticeably faster than bash.

• There are also some other minor things that I like in ksh93 like the typeset command that provides useful features like converting from upper case text to lower case. 

Of course nothing is perfect and there are some minor things that are missing in ksh93:

• no !$ substitute ($_ seems to be the ksh equiv)
   
• Arrow keys for navigating history. Actually ksh93 supports arrow keys starting with the 'h' point release.
   
• Tab for file completion. Works now, but only starting with  the 'h' point release. Older version of ksh93 used Esc \ or Esc=  to provide file completion. See KSH-93 - Frequently Asked Questions
   
• Incremental command completion. In ksh93 command completion works on the first word of command. ksh93 will do completion using aliases, functions, and commands (you can also do history search with /<prefix>Esc but it's less convenient).

  C-shell has a better feature called an incremental searching, usually bound to C-r. Press C-r and start typing a word which exists somewhere in the history. By the time you've typed enough of it to uniquely identify it, you've plucked the whole of that line out of the history. This is available in bash (C-r by default), tcsh (not bound by default) and zsh (C-r by default).
   

• The lack of cd history (even in its C-shell dirstack surrogate incarnation.) The directory stack is a questionable C-shell feature, but it should be considered as an early attempt to provide cd history in addition to command history and should be recognized as such. That's probably why despite its weaknesses it became rather popular and is now implemented in bash and zsh. And nothing succeeds like success. Typically there are three primitives: pushd, popd (usually bound to + and - keys) and dirs. Built-in pushd without parameters push the current directory into the stack. With a parameter it is similar to cd and it usually is used by defining it as an alias to cd  (alias cd='pushd ').   Build-in popd generalize cd - (return to prev directory) and permits to jump to an arbitrary directory in the stack.  dirs can print, store in a file, restore and clear the directory stack at any time. There are also the savedirs and dirsfile shell variables that can be set to store the directory stack automatically on logout and restore it on login. The dirstack shell variable can be examined to see the directory stack and set to put arbitrary directories into the directory stack.

  The best known workaround for ksh93 is to implement those built-ins as functions. See, for example,  a nice  reimplementation is described in the Learning the Korn shell book.  With some programming skills it actually can be made more user friendly then the current bash implementation ;-), but the inability to use  ~+digit and ~-digit  shortcuts everywhere still is a drawback ( cd ~+  can be imitated by cd - in ksh).
   

• The ability to use +/- with ~ as in bash and zsh. This feature is best described in the zsh documentation (note that in zsh the stack is 1-based, with the current directory treated as the 0th element):

  Each word is checked to see if it begins with an unquoted `~'.  If it  does, then the word up to a `/', or the end of the word if there is no  `/', is checked to see if it can be substituted in one of the ways
  described here.  If so, then the `~' and the checked portion are  replaced with the appropriate substitute value.

   A `~' by itself is replaced by the value of $HOME.  A `~' followed by a  `+' or a `-' is replaced by the value of $PWD or $OLDPWD, respectively.

   A `~' followed by a number is replaced by the directory at that  position in the directory stack.  `~0' is equivalent to `~+', and `~1'   is the top of the stack.  `~+' followed by a number is replaced by the  directory at that position in the directory stack.  `~+0' is equivalent  to `~+', and `~+1' is the top of the stack.  `~-' followed by a number  is replaced by the directory that many positions from the bottom of the  stack.  `~-0' is the bottom of the stack.  The PUSHD_MINUS option  exchanges the effects of `~+' and `~-' where they are followed by a  number.

   A `~' followed by anything not already covered is looked up as a named  directory, and replaced by the value of that named directory if found.  Named directories are typically home directories for users on the  system.  They may also be defined if the text after the `~' is the name  of a string shell parameter whose value begins with a `/'.  It is also  possible to define directory names using the -d option to the hash  built-in.

Still I prefer ksh93 to bash for shell programming because of its better supports of pipes. A very cool feature of ksh93 is `|&` which is a way of getting a pipe to a background process (co-processes). You can also call it a bi-directional pipe. You can then use `read` and `print` to send and receive stuff from the  process started with `|&`. For example, if I need to do floating point math, I can start `bc` in the background with `|&`. Interesting. with zsh, |& is a pipe which takes both stdout and stderr, similar to >& and >>& for redirection. It's handy, especially when you need to use a program with a rather verbose --help option. Bash doesn't recognize |& at all :-(.

But the main shortcoming of bash it its treatment of the last stage of the pipe. Consider the following example derived from Slashdot discussion  of David Korn interview:

x="hello"
echo "goodbye" | read x
echo $x

In both bash and pdksh, running the script would incorrectly return "hello", because the "read x" would be run in a subshell with its own environment. The subshell would then close without affecting the instance of x in the main shell. That's suck. In ksh, however, putting 'read' at the end of a pipeline does not create a subshell, so running the script would return "goodbye" just as any programmer might expect. Scripts that rely on classic Unix shell idiom "echo $string | read x y junk" are convenient to parse space-separated lists, and of course these statements will never work in bash (you need to do a few backtick/sed operations). Actually, the best way to do this bash is the following kludge. You need to save off the current argument list ($*) if necessary, and then:

set -- $string
x=$1; shift
y=$1; shift
junk="$*"

As you can see the ksh implementation is much cleaner and elegant.

The example above is a small, but nice illustration of my main point: one of the most important concepts traditionally associated with shells is piping -- I urge you to learn this concept as deeply as possible -- this is actually a pretty powerful programming paradigm and many classic software products, including such complex as compilers,  can re restructured to a solution my several stages of pipes (coroutines were actually invented by a compiler writer). Unfortunately compiler construction it now out of fashion and seldom taught in nowadays university curriculums.

Again I feel that sub par support of piping is the major weakness of Perl that prevented it from overtaking the Unix scripting despite unique favorable historic conditions (Perl became the first Web scripting language).

This page emphasizes ksh93 which I think is the most underappreciated shell in Unix world.  But I am not a shell fundamentalist and bash is pretty much OK as a login shell and it is somewhat more user friendly as a command line shell (please note that the recent versions of ksh93 support arrow-based navigation of history and TAB-based file completion).

For an long time, beginners were better off from starting with tcsh that is the most user friendly shell out of the box. But right now bash  (v 3.2 and later) is probably competitive to tcsh as for the user friendliness, so why bother. Most user friendly features introduced by csh like 'autolist' now have their counterparts in bash too and the latest versions of bash support up/down browsing of command history (the hallmark of csh and tcsh). Command completion mode is now also borrowed from tcsh -- all variants can be listed after an ambiguous completion by pressing tab (bash requires two tab's), e.g.:

 % ls /usr/l[tab] lib/ libexec/ local/.

In bash this can be set via readline. In /etc/inputrc or ~/.inputrc, put "set show-all-if-ambiguous on". This'll cause bash (which uses readline) to list all possibilities after a single tab. Check the readline man page for other nifty stuff, including the incremental reverse-search-history (default Ctrl-R).

Another important point. Shell is far from being perfect for navigation in a complex filesystem. It does have a rudimentary features like pushd/popd, but there are not enough. I've never see a Unix administrator who is able to complete in speed and productivity with sysadmin who is an expert user of a good orthodox file manager. There are several open source implementations of such manages.  I recommend to use Midnight Commander -- see my OFM page. It's more or less feature-complete and is OK interactive shell for Linux and other Unixes. A little bit heavy for remote usage, but with the current line speed still OK. Some versions are pretty buggy. From 2000 to 2002 Midnight Commander suffered from the desire to turn it into a graphical gnome file manager, but later Pavel Roshkin managed to straiten things up and remove most of the gnome-related code (he preserved the dependence on glib, though).  See my on-line book about Orthodox file managers.

Other useful links:  Editors Links,  Unix/Linux links. Overview of books about shells is in Shell bibliography. Again, a highly recommended shell site is SHELLdorado.  You should probably visit it first. More or less useful dotfiles can be found at dotfiles.com home. Check it out.

Dr. Nikolai Bezroukov

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Created May 16, 1996; Last modified: March 12, 2019