IFS variable and Field splitting in the Korn shell and Bash
System variable IFS specifies internal field separators (normally space,
tab, and new line) used to separate command words that result from command or parameter
substitution and for separating words that are read using built-in command read.
The first character of the IFS parameter is used to separate arguments
for the $* substitution.
After performing command substitution, the Korn shell scans the results of substitutions
for those field separator characters found in the IFS (Internal Field Separator)
variable. Where such characters are found, the shell splits the substitutions into
distinct arguments.
The shell retains explicit null arguments ("" or '') and removes implicit null arguments
(those resulting from parameters that have no values).
- If the value of IFS is a space, tab, or newline character, or if
it is not set, any sequence of space, tab, or newline characters at the beginning
or end of the input will be ignored and any sequence of those characters within
the input will delimit a field. For example, the following input yields two
fields, school and days:
<newline><space><tab>school<tab><tab>days<space>
-
If the value of IFS is not null, the following rules apply in sequence.
IFS white space is used to mean any sequence (zero
or more instances) of white-space characters that are in the IFS value
(for example, if IFS contains space, comma and tab, then any sequence
of space and tab characters is considered IFS white space).
- IFS white space is ignored at the beginning and end of the
input.
- Each occurrence in the input of an IFS character that is not
IFS white space, along with any adjacent IFS white space,
delimits a field.
- Nonzero length IFS white space delimits a field.
-
Why are the elements of "$*" are separated by the first character of
IFS instead of just spaces? To give you output flexibility. As a simple
example, let's say you want to print a list of positional parameters separated
by commas. This script would do it:
IFS=','
print $*
Changing IFS in a script is fairly risky, but it's probably OK as long
as nothing is called from the the script that depends on the value of IFS.
If this script were called arglist, then the command arglist bob dave
ed would produce the output bob,dave,ed.
October 13, 2003
... The shell uses the value stored in IFS, which is the space, tab, and
newline characters by default, to delimit words for the read and set commands,
when parsing output from command substitution, and when performing variable
substitution.
IFS can be redefined to parse one or more lines of data whose fields are
not delimited by the default white-space characters. Consider this sequence
of variable assignments and for loops:
line=learn:unix:at:livefire:labs
... ... ...
OIFS=$IFS
$ IFS=':'
for i in $line; do; echo $i; done
learn
unix
at
livefire
labs
The first command assigns the string "learn:unix:at:livefire:labs" to the
variable named line. You can see from the first for loop that the shell treats
the entire string as a single field. This is because the string does not contain
a space, tab, or newline character.
After redefining IFS, the second for loop treats the string as four separated
fields, each delimited by a colon. Using a colon for IFS would be appropriate
when parsing the fields in a record from /etc/passwd, the user account information
file:
livefire:x:100:1::/export/home/livefire:/bin/ksh
Notice that the original value of IFS was stored in OIFS ("O" for original)
prior to changing its value. After you are finished using the new definition,
it would be wise to return it to its original value to avoid unexpected side
effects that may surface later on in your script.
TIP – The current value of IFS may be viewed using the following pipeline:
echo "$IFS" | od -b
0000000 040 011 012 012
0000004
$
The output of the echo command is piped into the octal dump command, giving
you its octal equivalent. You can then use an ASCII table to determine what
characters are stored in the variable. Hint: Ignore the first set of zeros and
the second newline character (012), which was generated by echo.
Korn
Shell Unix and Linux Programming Manual, Third Edition
Substitution
The first step the shell takes in executing a simple-command is to perform
substitutions on the words of the command. There are three kinds of substitution:
parameter, command and arithmetic. Parameter substitutions, which are described
in detail in the next section, take the form $name or ${...}; command substitutions
take the form $(command) or 'command'; and arithmetic substitutions take the
form $((expression)). If a substitution appears outside of double quotes, the
results of the substitution are generally subject to word or field splitting
according to the current value of the IFS parameter. The IFS parameter specifies
a list of characters which are used to break a string up into several words;
any characters from the set space, tab and newline that appear in the IFS characters
are called IFS white space. Sequences of one or more IFS white space characters,
in combination with zero or one non-IFS white space characters delimit a field.
As a special case, leading and trailing IFS white space is stripped (i.e., no
leading or trailing empty field is created by it); leading or trailing non-IFS
white space does create an empty field. Example: if IFS is set to ':', the sequence
of characters 'A:B::D' contains four fields: 'A', 'B', '' and 'D'. Note that
if the IFS parameter is set to the null string, no field splitting is done;
if the parameter is unset, the default value of space, tab and newline is used.
Here is a script that looks for ^ in shell scripts in every directory
in your PATH:
[2] The exact message varies from system to system; make sure that yours
prints this message when given the name of a shell script. If not, just
substitute the message your file command prints for "shell script"
in the code below.
IFS=:
for d in $PATH; do
print checking $d:
cd $d
scripts=$(file * | grep 'shell script' | cut -d: -f1)
for f in $scripts; do
grep '' $f /dev/null
done
done
The first line of this script make it possible to use $PATH as an
item list in the for loop. For each directory, it cds there
and finds all shell scripts by piping the file command into grep
and then, to extract the filename only, into cut. Then for each shell
script, it searches for the ^ character. [3]
[3] The inclusion of /dev/null in the grep command
is a kludge that forces grep to print the names of files that contain
a match, even if there is only one such file in a given directory.
Extra tidbit: I recently found a quirky difference between Korn shell ('93)
and bash. Consider the following:
echo foo | read bar; echo $bar
... whenever you see a "|" operator in a shell command sequence you should understand
that there is implicitly a subshell (new process)
that is created (forked) on one side of it or the other.
Of course other processes (including subshells) cannot affect the values
of your shell variables. So the sequence above consists of three commands (echo
the string "foo", read something and assign it to a shell variable named "bar",
and echo the value of (read the $ dereferencing operator as "the value of")
the shell named "bar"). It consists of two processes. One on one side of the
pipe, and the other on the other side of the pipe. At the semicolon the shell
waits for the completion of any programs and commands that precede it, and then
continues with a new command sequence in the current shell.
The question becomes whether the subshell was created on the left or the
right of the | in this command. In bash it is clearly created on the right.
The 'read' command executes in a subshell. That then exits (thus "forgetting"
its variable and environment heaps). Thus $bar is unaffected after the semicolon.
In ksh '93 and in zsh the subshell seems to be created to the left of the
pipe. The 'read' command is executed in the current shell and thus the local
value of "bar" is affected. Then the subsequent access to that shell variable
does reflect the new value.
As far as I know the POSIX spec is silent on this point. It may even be that
ksh '93 and zsh are in violation of the spec. If so, the spec is wrong!
It is very useful to be able to parse a set of command outputs into a local
list of shell variables. Note that for a single variable this is easy:
bar=$(echo foo)
or: bar=`echo foo` ... are equivalent expressions and they work just
fine.
However, when we want to read the outputs into several values, and especially
when we want to do so using the IFS environment value to parse these values
then we have to resort of inordinate amounts of fussing in bash while ksh '93
and newer versions of zsh allow us to do something like:
grep ^joe /etc/passwd | IFS=":" read login pw uid gid gecos home sh
(Note the form: 'VAR=val cmd' as shown here is also a bit obscure but handy.
The value of VAR is only affected for the duration of the following command
--- thus saving us the trouble of saving the old IFS value, executing
our 'read' command and restoring the IFS).
BTW: If you do need to save/restore something like IFS you must using proper
quoting. For example:
OLDIFS="$IFS"
# MUST have double/soft quotes here!
IFS=:,
# do stuff parsing words on colons and commas
IFS="$OLDIFS"
# MUST also have double/soft quotes here!
Anyway, I would like to do some more teaching in the field of shell scripting.
I also plan to get as good with C and Python as I currently am with 'sh'. That'll
take at least another year or so, and a lot more practice!
Others have showed you:
IFS='
' # actual newline in quotes
My usual idiom for what you're doing is:
ps | while read nxt; do
...
done
But there's a gotcha here. The Bourne shell forks a separate shell for
a set of shell commands that are being piped to. It does a decent job
of hiding this fact, but your:
eval jbid_$dex=$e
IFS Specifies internal field separators (normally space, tab, and new
line) used to separate command words that result from command or parameter substitution
and for separating words with the regular built-in command read. The
first character of the IFS parameter is used to separate arguments
for the $* substitution.
The prompt can be specified in the read statement:
$ read var?prompt
If var is not defined, input is assigned to variable REPLY. Field separator
can be assigned with the IFS ( Internal Field Separator) variable.
Example:
> cat kpwd
#!/bin/ksh
#-----------kpwd: read example in Korn shell
echo Proc $0: type pwd info with Korn shell
echo
read ok?"Type pwd info? (y/n)" #read with prompt
[[ $ok = @([Nn])* ]] && exit 1 #test read variable
echo pwd data are:
echo ""
IFS=: #set IFS to :
exec 0</etc/passwd #redirect stdin to /etc/passwd
# list users
#
while read -r NAME PAS UID GID COM HOME SHELL
do
print "acct= $NAME - home= $HOME - shell= $SHELL:"
done
#----------end script------------------
> kpwd
Type pwd info? (y/n)y
pwd data are:
acct= john - home= /home/john - shell= /bin/tcsh:
acct= mary - home= /home/mary - shell= /bin/tcsh:
acct= tester - home= /d4/check - shell= /bin/sh:
>
If you need to run a script in the current shell either use the dot [.] or the
source command:
$ . .profile
$ source .profile
$ rehash
In this way any variable, alias or function setting stay in effect. Note the
rehash command that recreates the in-memory shell tables and grants that the
system acknowledges new .profile definitions.
Society
Groupthink :
Two Party System
as Polyarchy :
Corruption of Regulators :
Bureaucracies :
Understanding Micromanagers
and Control Freaks : Toxic Managers :
Harvard Mafia :
Diplomatic Communication
: Surviving a Bad Performance
Review : Insufficient Retirement Funds as
Immanent Problem of Neoliberal Regime : PseudoScience :
Who Rules America :
Neoliberalism
: The Iron
Law of Oligarchy :
Libertarian Philosophy
Quotes
War and Peace
: Skeptical
Finance : John
Kenneth Galbraith :Talleyrand :
Oscar Wilde :
Otto Von Bismarck :
Keynes :
George Carlin :
Skeptics :
Propaganda : SE
quotes : Language Design and Programming Quotes :
Random IT-related quotes :
Somerset Maugham :
Marcus Aurelius :
Kurt Vonnegut :
Eric Hoffer :
Winston Churchill :
Napoleon Bonaparte :
Ambrose Bierce :
Bernard Shaw :
Mark Twain Quotes
Bulletin:
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient
markets hypothesis :
Political Skeptic Bulletin, 2013 :
Unemployment Bulletin, 2010 :
Vol 23, No.10
(October, 2011) An observation about corporate security departments :
Slightly Skeptical Euromaydan Chronicles, June 2014 :
Greenspan legacy bulletin, 2008 :
Vol 25, No.10 (October, 2013) Cryptolocker Trojan
(Win32/Crilock.A) :
Vol 25, No.08 (August, 2013) Cloud providers
as intelligence collection hubs :
Financial Humor Bulletin, 2010 :
Inequality Bulletin, 2009 :
Financial Humor Bulletin, 2008 :
Copyleft Problems
Bulletin, 2004 :
Financial Humor Bulletin, 2011 :
Energy Bulletin, 2010 :
Malware Protection Bulletin, 2010 : Vol 26,
No.1 (January, 2013) Object-Oriented Cult :
Political Skeptic Bulletin, 2011 :
Vol 23, No.11 (November, 2011) Softpanorama classification
of sysadmin horror stories : Vol 25, No.05
(May, 2013) Corporate bullshit as a communication method :
Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
History:
Fifty glorious years (1950-2000):
the triumph of the US computer engineering :
Donald Knuth : TAoCP
and its Influence of Computer Science : Richard Stallman
: Linus Torvalds :
Larry Wall :
John K. Ousterhout :
CTSS : Multix OS Unix
History : Unix shell history :
VI editor :
History of pipes concept :
Solaris : MS DOS
: Programming Languages History :
PL/1 : Simula 67 :
C :
History of GCC development :
Scripting Languages :
Perl history :
OS History : Mail :
DNS : SSH
: CPU Instruction Sets :
SPARC systems 1987-2006 :
Norton Commander :
Norton Utilities :
Norton Ghost :
Frontpage history :
Malware Defense History :
GNU Screen :
OSS early history
Classic books:
The Peter
Principle : Parkinson
Law : 1984 :
The Mythical Man-Month :
How to Solve It by George Polya :
The Art of Computer Programming :
The Elements of Programming Style :
The Unix Hater’s Handbook :
The Jargon file :
The True Believer :
Programming Pearls :
The Good Soldier Svejk :
The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society :
Ten Commandments
of the IT Slackers Society : Computer Humor Collection
: BSD Logo Story :
The Cuckoo's Egg :
IT Slang : C++ Humor
: ARE YOU A BBS ADDICT? :
The Perl Purity Test :
Object oriented programmers of all nations
: Financial Humor :
Financial Humor Bulletin,
2008 : Financial
Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related
Humor : Programming Language Humor :
Goldman Sachs related humor :
Greenspan humor : C Humor :
Scripting Humor :
Real Programmers Humor :
Web Humor : GPL-related Humor
: OFM Humor :
Politically Incorrect Humor :
IDS Humor :
"Linux Sucks" Humor : Russian
Musical Humor : Best Russian Programmer
Humor : Microsoft plans to buy Catholic Church
: Richard Stallman Related Humor :
Admin Humor : Perl-related
Humor : Linus Torvalds Related
humor : PseudoScience Related Humor :
Networking Humor :
Shell Humor :
Financial Humor Bulletin,
2011 : Financial
Humor Bulletin, 2012 :
Financial Humor Bulletin,
2013 : Java Humor : Software
Engineering Humor : Sun Solaris Related Humor :
Education Humor : IBM
Humor : Assembler-related Humor :
VIM Humor : Computer
Viruses Humor : Bright tomorrow is rescheduled
to a day after tomorrow : Classic Computer
Humor
The Last but not Least Technology is dominated by
two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt.
Ph.D
Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org
was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP)
without any remuneration. This document is an industrial compilation designed and created exclusively
for educational use and is distributed under the Softpanorama Content License.
Original materials copyright belong
to respective owners. Quotes are made for educational purposes only
in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains
copyrighted material the use of which has not always been specifically
authorized by the copyright owner. We are making such material available
to advance understanding of computer science, IT technology, economic, scientific, and social
issues. We believe this constitutes a 'fair use' of any such
copyrighted material as provided by section 107 of the US Copyright Law according to which
such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free)
site written by people for whom English is not a native language. Grammar and spelling errors should
be expected. The site contain some broken links as it develops like a living tree...
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or
referenced source) and are
not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness
of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be
tracked by Google please disable Javascript for this site. This site is perfectly usable without
Javascript.
Last modified:
July, 28, 2019
