KSH Substitutions

For in depth coverage of shell string manipulation capabilities see String Operations in Shell

The curly-bracket syntax allows for the shell's string operators. String operators in shell use unique among programming language curly-bracket syntax In particular, string operators let you do the following:

• Determine the length of the string and extract substring from the string
• Ensure that variables exist (i.e., are defined and have non-null values) and set default values for variables and catch errors that result from variables not being set
• Perform basic pattern matching

In shell any variable can be displayed as ${foo} without changing its meaning. This notation was extended to allow expressions inside curvy brackets., for example ${foo=moo}. Each operation is encoded using special symbol or two symbols ("digram", for example :-, :=, etc) . An argument that the operator may need is posited after the symbol of the operation.

The first group of string-handling operators tests for the existence of variables and allows substitutions of default values under certain conditions. These are listed in

Note: The colon (:) in each of these operators is actually optional. If the colon is omitted, then change "exists and isn't null" to "exists" in each definition, i.e., the operator tests for existence only.

Any variable can be displayed as ${foo} without changing its meaning. This functionality often is used to protect a variable name from surrounding characters.

$ export foo=foo 

$ echo ${foo}bar # foo exists so this works as expected

foobar

$ echo $foobar # foobar doesn't exist, so this doesn't

Three kinds of variable substitution are available for use: pattern matching, substitution and command substitution. I talk about the first two variables here and leave command substitution for another time.

Note:  Although the # and % operators may not seem obvious, they have a convenient mnemonic. The # key is on the left side of the $ key and operates from the left. The % key is on the right of the $ key and operates from the right. (This is true, at least, for US qwerty keyboards.)

Length Operator

One is ${#varname} returns the length of the value of the variable as a character string.

var="Hello world"
echo ${#var} 

Substr

It's useful to be able to take strings apart and put them together in different ways. Here is how to select a substring from a string:

string="this is a test"
substring=${string:1:9}              

In this example, the variable "substring" contains the word "substring". Remember this rule:

substring=${string_variable_name:starting_position:length}

The string starting position is zero-based.

Substitution Operators (Default value assignments)

Substitution operators mainly allow to check if the variable has null value of is undefined and return default value with or without assignment of this value to the variable. Here are major types fo substitution operators:

NOTE:  The ":" in the above digrams can be omitted. Such operator simply  tests for the existence of the variable.

Operator: ${foo:-bar}

Function: If $foo exists and is not null, return $foo. If it doesn't exist or is null, return bar.

Example:

$ export foo=""

$ echo ${foo:-one}

one

$ echo $foo

More complex example:

sort -nr $1 | head -${2:-10}

A typical usage include situations when you need to check if arguments were passed to the script and if not assign some default values::

#!/bin/bash 

export FROM=${1:-"~root/.profile"} 

export TO=${2:-"~my/.profile"}

cp -p $FROM $TO

Operator: ${foo:=bar}

Function: If $foo exists and is not null, return $foo. If it doesn't exist or is null, set $foo to bar and return bar.

Example:

$ export foo=""
$ echo ${foo:=one}
one

Operator: ${foo:+bar}

Function: If $foo exists and is not null, return bar. If it doesn't exist or is null, return a null.

Example:

$ export foo="this is a test"
$ echo ${foo:+bar}
bar

Operator: ${foo:?"error message"}

Function: If $foo exists and isn't null, return its value. If it doesn't exist or is null, print the error message. If no error message is given, it prints parameter null or not set. In a non-interactive shell, this aborts the current script. In an interactive shell, this simply prints the error message.

filename=${1:?"filename missing."}

Example:

$ export foo="one"

$ for i in foo bar baz; do

> eval echo \${$i:?}

> done

one

bash: bar: parameter null or not set

bash: baz: parameter null or not set 

 This, in turn, causes the creation of a variable, for example:

$ export foo="this is a test"
$ echo $bar

$ echo ${foo=bar}
this is a test
$ echo ${bar=bar}
bar
$ echo $bar
bar

Pattern Matching

Pattern-matching operators were introduced in ksh88 in a very idiosyncratic way. The notation is different from used by  Perl or utilities such as grep. That's a shame, but that's how it it. Life is not perfect. They are hard to remember, but there is a handy mnemonic tip: #  matches the front because number signs precede  numbers; %  matches the rear because percent signs follow  numbers.

The classic use for pattern-matching operators is stripping off components of pathnames, such as directory prefixes and filename suffixes. With that in mind, here is an example that shows how all of the operators work. Assume that the variable path  has the value /home /billr/mem/long.file.name; then:

Expression Result
${path##/*/}                       long.file.name
${path#/*/}              billr/mem/long.file.name
$path              /home/billr/mem/long.file.name
${path%.*}         /home/billr/mem/long.file
${path%%.*}        /home/billr/mem/long

Operator #:  ${foo#t*is}  deletes the shortest possible match from the left

Example:

$ export foo="this is a test"

$ echo ${foo#t*is}

is a test

 

Operator ##: ${foo##t*is} deletes the longest possible match from the left

Example:

$ export foo="this is a test"

$ echo ${foo##t*is}

a test

 

Operator %: ${foo%t*st} Function: deletes the shortest possible match from the right

Example:

$ export foo="this is a test"

$ echo ${foo%t*st}

this is a

for i in *.htm*; do

if [ -f ${i%l} ]; then 

echo "${i%l} already exists"

else 

mv $i ${i%l}

fi 

done

Operator %%: ${foo%%t*st} deletes the longest possible match from the right

Example:

$ export foo="this is a test"

$ echo ${foo%%t*st}

Shell Regular expressions

Think of regular expressions as strings that match patterns more powerfully than the standard shell wildcard schema. Regular expressions began as an idea in theoretical computer science, but they have found their way into many nooks and crannies of everyday, practical computing. The syntax used to represent them may vary, but the concepts are very much the same.

A shell regular expression can contain regular characters, standard wildcard characters, and additional operators that are more powerful than wildcards. Each such operator has the form x(exp), where x  is the particular operator and exp  is any regular expression (often simply a regular string). The operator determines how many occurrences of exp  a string that matches the pattern can contain.

The following section compares Korn shell regular expressions to analogous features in awk  and egrep. If you aren't familiar with these, skip to the section entitled "Pattern-matching Operators."

shell basic regex vs awk/egrep regular expressions

These equivalents are close but not quite exact. Actually, an exp  within any of the Korn shell operators can be a series of exp1|exp2|... alternates. But because the shell would interpret an expression like dave|fred|bob  as a pipeline of commands, you must use @(dave|fred|bob)  for alternates

For example:

• @(dave|fred|bob)  matches dave, fred, or bob.
• *(dave|fred|bob)  means, "0 or more occurrences of dave, fred, or bob". This expression matches strings like the null string, dave, davedave, fred, bobfred, bobbobdavefredbobfred, etc.
• +(dave|fred|bob)  matches any of the above except the null string.
• ?(dave|fred|bob)  matches the null string, dave, fred, or bob.
• !(dave|fred|bob)  matches anything except dave, fred, or bob.

It is worth re-emphasizing that shell regular expressions can still contain standard shell wildcards. Thus, the shell wildcard ?  (match any single character) is the equivalent to .  in egrep  or awk, and the shell's character set operator [...]  is the same as in those utilities.  For example, the expression +([0-9])  matches a number, i.e., one or more digits. The shell wildcard character * is equivalent to the shell regular expression * (?).

A few egrep  and awk  regexp operators do not have equivalents in the Korn shell. These include:

• The beginning- and end-of-line operators ^  and $.
• The beginning- and end-of-word operators \<  and \>.
• Repeat factors like \{N  \}  and \{M  ,  N  \}.

The first two pairs are hardly necessary, since the Korn shell doesn't normally operate on text files and does parse strings into words itself.

NEWS CONTENTS

Old News ;-)

Recommended Links

Softpanorama Recommended

Top articles

Sites

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: September 04, 2019