|
|
Home | Switchboard | Unix Administration | Red Hat | TCP/IP Networks | Neoliberalism | Toxic Managers |
| (slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix | |||||||
| Bulletin | 1998 | 1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 |
| 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 |
| Jan | Feb | Mar | Apr | May | Jun | Jul | Sept | Oct | Nov | Dec |
|
|
|
|
Complex regex are constructed from simple regular expressions using the following metacharacters:
Perl provides several capability to specify how many times a given component must be present before the match is true. You can specify both minimum and maximum number of repetitions.
One can see that old quantifiers that we already know (*, + and ?) can be expressed via new ones:
m/^\s*\w+/;
| Be careful when using the * quantifier because it can match an empty string, which might not be your intention. The regex /b*/ will match any string - even one without any b characters. |
At times, you may need to match an exact number of components. The following match statement will be true only if five words are present in the $_ variable:
$_ = 131.1.1.1 - joejerk [21/Jan/2000:09:50:50 -0500] "GET http://216.1.1.1/xxxgirls/bigbreast.gif HTTP/1.0" 200 51500
m/(\w+\s+){3}/; # get the user name of the offender
In this example, we are interested in getting exactly the third word which corresponds
to the user id in HTTP logs. After match $3 should contain this id.
The same ideas can be used for processing date and time in the HTTP logs.
The character class [0123456789] or, shorter, [0-9] defines the class of decimal digits, and [0-9a-fA-F] defines the class of hexadecimal digits. You should use a dash to define a range of consecutive characters. You can use metacharacters inside character classes ( but not as endpoints of a range). For example:
$test = "A\t12";
if ( m/[XYZ\s]/ ) {
print "Variable test matched the regex\n"
which will display
Variable test matched the regex
because the value of $test includes the tab character which matched metacharacter \s in the character class [XYZ\s].
| Meta-character . and modifiers ?, *, + that appear inside the square brackets that define a character class are used in their literal sense. They lose their meta-meaning. |
Alternation allow you to provide several alternative regex and only one of those would match for the success of regex. In other words, the regular expression:
/^foreach|^for|^while/
means "look for the line beginning with the string 'for' OR the string 'if' or the string 'while'."
The ( | ) syntax split regular expression on sections and each section will be tried independently. Alternation always tries to match the first item in the parentheses. If it doesn't match, the second pattern is then tried and so on.
This is called left most matching, and it is very similar to short-circuit operator ||. Misunderstand the fact that evaluation of alternative stops at the first success can lead to subtle bugs if one string in alternation is a substring of the other:
$line = 'foreach $i (@n) { $sum+=$i;}';
if ($line =~ /^for|^while/ ) {
print "Regular loop\n";
} elsif ( $line =~ ^foreach/ ) {
print "Foreach loop\n");
}
In this case the string foreach will never be matched as string for will match before it. This is a common a mistake and it is prudent always put longest string first in such cases. This tip is also helpful when you don't know whether or not a word will be followed by a delimiter, or an end of line character, or whether or not a word is plural:
for $line ('words', 'word') {
if ($line =~ /\bwords\b/ ) {
print "singular\n";
} elsif ($line =~/\bword\b/ ) {
print "Plural\n";
}
In general longest match should be first
The useful modifier for matching is modifier i (ignore case). for example
$line =~ /"word(s?)/iwill match "word" or "words" independent of case.
If some part of regex is enclosed in parenthesis it is considered a group and matching to this groups substring is assigned to special variables $1, $2,.... For example:
$ip='10.192.10.1'; $ip=~/(\d+)\.(\d+)\.(\d+)\.(d+)/; Print "Ip address components are $1, $2, $3 and $4\n";
You can use alteration within the group, for example
(red|blue)If regular expression matched the string, the substrings that matched each group are assigned to so called "capture variables": $1, $2, $3, ... In other words each group captures what it content matche and assign it to corresponding capture variable.
One important feature of capture variables is that only successful match affect them. If match is unsuccessful, then previous values are preserved whatever they may be. That leads to difficult to find errors if you are not careful. You should never use capture variables without checking if the march is successful or not.
For example the regex /\w+/ will let you determine if $_ contains a word, but does not let you know what the word is. In order to accomplish that, you need to enclose the matching components with parentheses. For example:
if ( m/(\w+)/ ) {
$word=$1;
}
By doing this, you force Perl to store the matched string into the $1 variable. The $1 variable can be considered as pattern memory or backreference. We will discuss backreferences in more details later.
As well as identifying substrings that match regular expressions Perl can make substitutions based on those matches. The way to do this is to use the s function which mimics the way substitution is done in the vi text editor. If the target string is omitted then the substitution is assumed to take place with the $_ variable.
To replace an occurrence of regular expression h.*?o by string Privyet; in the string $sentence we use the expression
$sentence =~ s/h.*?o/Privyet/;
and to do the same thing with the $_ variable just write the right side of the previous operator:
s/h.*?o/Privyet/;
The first part of this expression is called matching pattern and the second part is called substitution string. The result of a substitution operator in the scalar context is the number of substitutions made, so it is either 0 (false) or 1 (true) in this case.
|
The result of a substitution operator in the scalar context is the number of substitutions made |
This example only replaces the first occurrence of the string, and it may be that there will be more than one such string we want to replace. To make a global substitution the last slash is followed by a g modifier as follows:
s/h.*?o/Privyet/g
Here the target is $_ variable. The expression returns the number of substitutions made ( 0 is none).
If we want to also make replacements case insensitive with the modifier i (for "ignore case"). The expression
s/h.*?o/Privyet/gi
will force the regex engine to ignoring case. Note that case will be ignored only in matching -- substitution string will in inserted exactly as you specified.
|
The substitution operator can also be used to delete any substring. In this case the replacement string should be omitted. For example to remove the substring "Nick" from the $_ variable, you could write: s/Nick//;
There is additional modifier that is applicable to both regex and replacement string -- /e modifier that changes the interpretation of the pattern delimiters. If used, variable interpolation is active even if single quotes are used.
Like in index function you can use variables in both matching pattern and substitution string. For instance:
# let's assume that $_ = "Nick Bezroukov"; $regex = "Nick"; $replacement_string = "Nickolas"; $result = s/$regex/$replacement_string/;
Here is a slightly more complex example of replacment (Snort rules):
#alert udp $site_dhcp 63 -> any any (msg:"policy tftp to dchp segment"; classtype:attempted-admin; sid:235; rev:60803;)
$new="classtype:$ARGV[0];";
while(<>) {
$line=$_;
$line=~s[classtype\:.*?\;][$new];
print $line;
}
This program changes the $_ variable by performing the replacement and the $result variable will be equal to 1 -- the number of substitutions made.
For a single substitution of a string a similar capability is available with built-in function substr.
$result = substr($_,index($_'Nick'),length('Nick'));
If would be nice to be able to match on array too but this is not the case. If you try something such as:
@buffer =~ m/yahoo/; # Wrong way to search for a string in the array
In the example above the array @buffer will be converted to scalar (number of its elements) and if we assume that the array has 10 elements that means that you will be doing something like:
'10' =~ m/yahoo/;
The right way to solve this problem is to use grep function like in an example below:
grep(m/variable/, @buffer);
In scalar context the number of matches will be returned. In array context the list of elements that matched will be returned.
Each matched group in matching pattern can be referenced with so called backreferences. Backreferences are also numbers consecutively \1, \2. \3. ... They can be used both in matching pattern and in replacement string.
With the "standard" notation you need to use a backslash to escape special character you want to match. That means that you still need to use the backslash character to escape any of the meta-characters including slash, which is pretty common in Unix pathnames. For example:
$path =~ m/usr\/local\/bin/;
This tries to match /usr/local/bin in $path. As we saw regular expressions are usually delimited by slashes and if regex that you want to match contain a lot of slashes the whole regular expression becomes unreadable
m/Hello/;
To rectify this problem Perl allows the use of alternative regex delimiters (delimiter that marks the beginning and end of a given regular expression) if you use initial m for matching:
m{/usr/local/bin/}
Actually { } is probably the most readable alternative variant that permit easy finding of opening and closing brackets in any decent editor (including Emacs, vi, vim, Slickedit, MultiEdit). But if you wish you can use other symbols, for example:
m"/usr/local/bin" # here double quote serves as a regex delimiter
Note that if a left bracket is used as the starting delimiter, then the ending delimiter must be the right bracket. Both the match and substitution operators let you use variable interpolation.
In case you regex contains a lot of special symbols you can first assign it to a single-quoted string and then use variable in the matching operator. The regex inside slashes are treated like double quoted strings and you can interpolate with them with a variable. For example:
$profile = '/root/home/.profile'; m/$profile/;
The same trick works for substitution too, for example:
s{USA}{Canada}sg;
This capability to find matching bracket can be useful when we deal with multiple line regular expressions using extension syntax described below.
If the match regex evaluates to the empty string, the last valid regex is used. So, if you see a statement like
if (//) {print;}
in a Perl program, look for the previous regular expression operator to see what the regex really is. The substitution operator also uses this interpretation of the empty regex (but never for the substitution part which is a string, not a regular expression).
Extended mode which is activated by using modifier x provides capability to write comments within the regex as well as use whitespace freely for readability. For example, instead of regular expression:
# Match an assignment like a=b;. $1 will be the name of the variable and the # first word. $2 will be the second word. m/^\s+(\w+)\W+(\w+)\s+$/;
We can write
m/^\s+ (?# leading spaces)
(w+) (?# get first word)
\s*=\s* (?# match = with white space before and after ignored )
(.*) (?# right part )
\; (?# final semicolon)
/x
Here we move groups to separate lines it improves readability and gives us opportunity
to put comments using asymmetrical brackets (?# and )But you can go too far and "kill with kindness". Here is an example of over-commented regular expression that is more difficult to read the one line version:
m/
(?# This regex will match any Unix style assignments in configuration file delimited with semicolon
(?# results are put into $1 and $2 if the match is successful.)
^ (?# Anchor this match to the beginning of the string)
\s* (?# skip over any whitespace characters)
(?# we use the * because there may be none)
(\w+) (?# Match the first word, put in the first variable)
(?# the first word because of the anchor)
\W+ (?# Match at least one non-word)
(?# character, there may be more than one)
(\w+) (?# Match another word, put into the second variable)
\s* (?# skip over any whitespace characters)
(?# use the * because there may be none)
$ (?# Anchor this match to the end of the)
(?# string. Because both ^ and $ anchors)
(?# are present, the entire string will)
(?# need to match the regex. A sub-string will not match.)
/x;
Please note that the really important trick of using \W to match any combination of delimited like "=" " = " or " =" remains unexplained. In a way those comments make regex more difficult to understand, not easier. In general, if you do not such an excesses. In commenting the first rule is not too much zeal ;-).
Along with the ability to add comments, suffix x also provides addition matching capabilities:
The most useful of the extensions listed above is grouping without creating a backreference.
You can also specify regex modifiers inside the regex itself
(?sxi)
This extension lets you specify an embedded modifier in the regex rather than adding it after the last delimiter. This is useful if you are storing regexs in variables and using variable interpolation to do the matching.
Extensions also let you change the order of evaluation without assigning the value of matched group to special variables ($1, $2,...). For example,
m/(?:Operator|Unix)+/;matches the strings Operator and Unix in any order. No special regex variables ($1, $2, $3,...) will be assigned.
At times, you might like to include a regex component in your regex without including it in the $& variable that holds the matched string. The technical term for this is a zero-width positive look-ahead assertion. You can use this to ensure that the string following the matched component is correct without affecting the matched value. For example, if you have some data that looks like this:
A:B:C
and you want to find all operators in /etc/passwd file and store the value of the first column, you can use a look-ahead assertion. This will do both tasks in one step. For example:
while (<>) {
push(@array, $&) if m/^\w+(?=\s+Operator\s+)/;
}
print("@array\n");
Let's look at the regex with comments added using the extended mode. In this case,
it doesn't make sense to add comments directly to the regex because the regex is
part of the if statement modifier. Adding comments in that location would
make the comments hard to format.
So we can use a different tactic and put the regex in variable
$ regex = '^\w+ (?# Match the first word in the string)
(?=\s+ (?# Use a look-ahead assertion to match)
(?# one or more whitespace characters)
Operator (?# text to match but not to include)
\s+' (?# one or more whitespace characters)
while (<>) {
push(@array, $&) if m/$ regex/xo;
}
print("@array\n");
Here we used a variable to hold the regex and then used variable interpolation in
the regex with the match operator. To speed things up we use o modifier, which tells
Perl to evaluate regular expression only once.
The last extension that we'll discuss is the zero-width negative assertion. This type of component is used to specify values that shouldn't follow the matched string. For example, using the same data as in the previous example, you can look for everyone who is not an operator. Your first inclination might be to simply replace the (?=...) with the (?!...) in the previous example.
| There are many ways of matching any value.
If the first method you try doesn't work, try breaking the value into smaller components and match each boundary. If all else fails, you can always ask for help on the comp.lang.perl.misc newsgroup. |
One of its more common uses of regex is find a substring in a string but remember that in simple cases the index function is simpler and better. Remember that regular expression matching is greedy and you will get the longest match possible:
$regex = "a*a";
$_ = "abracadabra";
if m/$regex/ {print "Found $regex in $_\n"
When matching lines in a file you can print matched strings along with their line
number using special variable $.
$target = "yahoo";
open(INPUT, "< visited_sites.dat");
while (<INPUT>) {
if (/$target/o ) {
print "Site $target was visited: $. $_";
}
}
close(INPUT);>
| The $. special variable keeps track of the record number. Every time the diamond operators read a line, this variable is incremented. |
Please note that this example would be better programmed using the index function.
So the question arise what is are additional capabilities of regexs that made them superior to string fuctions in complex situations. The answer is that regexs have so called matching memory or regex memory-- a set of special variables that are assigned values during matching operation for regexs a whole and each component of the regex enclosed inside parentheses. regex memory often called backreferences. This memory persists after the execution of a particular match statement. You can think about backreferences as a special kind of assignment statements.
Each time you use the parentheses '()' in regex Perl assumes that you want to assign the result of matching to a special variable with names like $1, $2, $3.... ). Naturally $1 can be used to refer to the first, $2 -- the second, $3 -- the third matched sub-pattern. These variables can then be accessed directly, by name, or indirectly by assigning the matching expression to an array.
You saw a simple example of this earlier right after the component descriptions. That example looked for the first word in a string and stored it into the first buffer, $1. The following small program
$_ = "a=5";
m/(\w+) = (\d+) /;
print("$1, $2);
will display
A, 5This is a simplified example of how one can process Unix-style configuration files. You can use as many buffers as you need. Each time you add a set of parentheses, another buffer is used. If you want to find all the word in the string, you need to use the /g match modifier. In order to find all the words, you can use a loop statement that loops until the match operator returns false.
$_ = "word1 word3 word3";
while (/(\w+)/g) {
print("$1\n");
}
Naturally, this program will display
word1 word2 word3because of each iteration exactly one new match will be printed. As you can see regex has internal memory and in case of using modifier g in the loop will continue extract parts of the the initial strings one by one. But much more interesting approach to a similar problem is to use array on the left side of the assignment statement:
$_ = "word1 word2 word3";
@matches = /(\w+)/g;
print("@matches\n");
The program will display:
word1 word2 word3To help you to know what matched and what did not Perl has several auxiliary built-in variables with really horrible names:
For example:
$text = "this matches 'THIS' not 'THAT'";
$text =~ m"('TH..')";
print "$` $'\n";
Here, Perl will save and later print substring "this matches '" for &` and "' not 'THAT'" for &'. the characters 'THIS' are printed out - Perl has saved them for you in $1 which later gets printed. That regular expressions match the first occurrence on a line. 'THIS' was matched because it came first. And, with the default regexp behavior, 'THIS' will always be the first string to be matched (you can change this default by modifiers - see below)
| If you need to save the value of the matched strings stored in the regex memory, make sure to assign them to other variables. regex memory is local to the enclosing block and lasts only until another match is done. |
backreferences are available in the matching regex itself. In other words, if you put parentheses around a group of characters, you can use this group of characters later in the regular expression or substitution string. But there is an important syntactic difference -- if you want to use the backreferences in the matching regex you need to use the syntax \1, \2, etc. If you want to use the backreferences in substitution string you use regular $1, $2, etc.
Perl is Perl and there are some irregularities in Perl regular expressions ;-) Here are some examples:
$line = 'Hello world'; $line =~s/(\w+) (\w+)/$2 $1/; # This makes string 'world Hello'.
We can also use of backreferenece in the matching regex itself:
if (/(A)(B)\2\1/) { print "Hello ABBA";}
The example is pretty artificial, but it well illustrates the key concept. There are 4 steps to the match of this string:
|
Note: If variable interpolation is used in the replacement string none of the meta-characters can be used in the replacement string |
Here are some more examples:
$text = 'word1 word2 word3'; ($word1, $word3) = ($text =~ m"(\w+).*(\w+)");
Notice, however, that assignment occurs when the text string matches. When the text string does not match, then $word1 and $word3 would be empty. Try the example above with the sting "1999 2000 2001" to see the result. So, what happens if your regular expression does not match at all? Nothing will be assigned and special variable will preserve thier values (so the values from prev match if any would be used).
|
Backreferences are not set if a regular expression fails |
This is a frequent Perl 'gotcha'. Built-in variables like $1 does not get change if the regular expression fails. Some people think this a bug, others consider this a feature. Nonetheless, this second point becomes painfully obvious when you consider the following code.
$_ = 'Perl bugs bite'; /\w+ (\w+) \w+/; # sets $1 to be "bugs". $_ = 'Another match another bug'; /(^a.*\s)"; # /^a.*\s will not match to any substring in the string print $1 # Suprize ! "bugs" will be printed !
In this case, $1 is the string 'bugs', since the second match expression failed! This Perl behavior can cause hours of searching for bug. So, consider yourself warned. Or more to the point, always check if a match was succcesful before assigning anything to it. You can use one of the following three checks to aviod this type of errors:
if (/(^a.*\s)/) {
$matched = $1;
} else {
print "matching failed";
}
($scalarName =~ m"(regular expression)") && ($match = $1);
($match1, $match2) = /(\w+).*(\w+));
if ($match1 eq '' || $match2 eq '' ) {
} else {
print " match failed\n"
}
Although the first method is the most clean any one will do the job. In any case your regex matching code should protect from unassigned built-in variable regex matching errors.
In any case your regex matching code should protect from unassigned built-in variable regex matching errors.
|
In any case your regex matching code should protect from unassigned built-in variable regex matching errors. |
There are several cases:
For example:
($variable, $equals, $value) = ($line =~ m"(\w+)\s*(=)\s*(\w+)");
This takes the first reference (\w+) and makes it $variable, the second reference (=) and makes it $equals, and the third reference (\w+) and makes it $value.
Another interesting case is Matching in array context, 'g' modifier. This takes the regular expression, applies it as many times as it can be applied, and then stuffs the results into an array that consists of all possible matches. For example:
$line = '1.2 3.4 beta 5.66'; @matches = ($line =~ m"(\d*\.\d+)"g);
will make '@matches' equal to '(1.2, 3.4, 5.66)'. The 'g' modifier does the iteration, matching 1.2 first, 3.4 second, and 5.66 third. Likewise:
undef $/;
my $FD = new FileHandle("file");
@comments = (<$FD> =~ m"/\*(.*?)\*/");
will make an array of all the comments in the file '$fd'
Finally, if you use the matching operator in scalar context, you get a behavior that is entirely different from anything else (in the regular expression world, and even the Perl world). This is that 'iterator' behavior we talked about. If you say:
$line = "BEGIN <data> BEGIN
<data2> BEGIN <data3>"
while ($line =~ m"BEGIN(.*?)(?=BEGIN|$)"sg){
push(@blocks, $1);
}
This then matches the following text (in bold), and stuffs it into @blocks on successive iterations of while:
BEGIN <data>(%)BEGIN <data2> BEGIN <data3> BEGIN <data> BEGIN <data2>(%)BEGIN <data3> >BEGIN <data> BEGIN <data2> BEGIN <data3>
We have indicated via a '(%)' where each of the iterations start their matching. Note the use of (?=) in this example too! It is essential to matching the correct way, since if you don't use it, the 'matcher' will get set in the wrong place.
As backreferences are implicit assignments they can be nested. Let's discuss parsing of date format in HTTP logs.
m{\([(\d)*\])};
Here, the outermost (( )) parentheses captures the whole thing: 'softly slowly surely subtly'. The innermost (()) parentheses captures a combination of strings beginning with an s and ending with a "ly" followed by spaces. Hence, it first captures 'softly', throws it away then captures 'slowly', throws it away then captures 'surely', then captures 'subtly'.
The first two examples are fairly straightforward. '[0-9]' matches the digit '1' in 'this has a digit (1) in it'. '[A-Z]' matches the capital 'A' in 'this has a capital letter (A) in it'. The last example is a little bit trickier. Since there is only one 'an' in the regex, the only characters that can possibly match are the last four 'an A'.
However, by asking for the regex 'an [^A]' we have distinctly told the regular expression to match 'a', then 'n', then a space, and finally a character that is NOT an 'A'. Hence, this does not match. If the regex was 'match an A not an e', then this would match, since the first 'an' would be skipped, and the second matched! Lik
$scalarName = "This has a tab( )or a newline in it so it matches";
$scalarName =~ m"[\t\n]" # Matches either a tab or a newline.
# matches since the tab is present
This example illustrates some of the fun things that can be done with matching and wildcarding. One, the same characters that you can have interpolated in a " " string also get interpolated in both a regular expression and inside a character class denoted by a brackets ([\t\n]). Here, "\t" becomes the matching of a tab, and "\n" becomes the matching of a newline.
m/a|b+/it's hard to tell if the regex should be
m/(a|b)+/ # match any sequence of "a" and "b" characters
# in any order.
or
m/a|(b+)/ # match either the "a" character or the "b" character
# repeated one or more times.
The order of precedence shown in below. By looking at the table, you can see that
quantifiers have a higher precedence than alternation. Therefore, the second interpretation
is correct.
| Precedence Level | Component |
|---|---|
| 1 | Parentheses |
| 2 | Quantifiers |
| 3 | Sequences and Anchors |
| 4 | Alternation |
| You can use parentheses to affect the order that components are evaluated because they have the highest precedence. you need to use extended syntax or you will be affecting the regex memory. |
Now let's introduce one new thing: both the matching and the substitution operators perform variable interpolation both in the regex and substitution strings, for example:
$variable =~ m"$scalar";
then $scalar will be interpolated, turned into the value for scalar. There is a caveat here. Any special characters will be acted upon by the regular expression engine, and may cause syntax errors. Hence if scalar is:
$scalar = "({";
Then saying something like:
$variable =~ m"$scalar";
is equivalent to saying: $variable =~ m"({"; which is a runtime syntax error. If you say:
$scalar = quotemeta('({');
instead will make $scalar become '\(\{' for you, and substitute $scalar for:
$variable =~ m"\{\{";
Then, you will match the string '({' as you would like.
You can use array in regex (it will be converted to the string with elements separated by spaces like in print statement), but this is tricky and rarely used:
$variable =~ m/@arrayName/; # this equals m/elem1 elem2/;
Here, this is equal to m/elem1 elem2/. If the special variable $" was set to '|', this would be equal to m/elem|elem2/, which as we shall see, matches either 'elem' or 'elem2' in a string. This works for special characters too:
For example:
$_ = "AAA BBB AAA"; print "Found bbb\n" if m/bbb/i;This program finds a match even though the regex uses lowercase and the string uses uppercase because the /i modifier was used, telling Perl to ignore the case. The result from a global regex match (modifier g) can be assigned to an array variable or used inside a loop.
As we already know the substitution operator has all modifiers used in the matching operator plus several more. One interesting modifier is the capability to evaluate the replacement regex as an expression instead of a string. You could use this capability to find all numbers in a file and multiply them by a given percentage. Or you could repeat matched strings by using the string repetition operator.
If back quotes are used as delimiters, the replacement string is executed as a DOS or UNIX command. The output of the command is then used as the replacement text.
e -- permits interpretation of the replacement part of the regular expression as a script
Without modifiers, a dot ('.') matches anything but a newline. Sometimes this is helpful. Sometimes it is very frustrating, especially if you have data that spans multiple lines. Consider the following case:
$line = 'BLOCK: Some text END BLOCK BLOCK: Another text END BLOCK'
Now suppose you want to match the text between keyword BLOCK and "END BLOCK":
$line =~ m{
BLOCK(\d+)
(.*?)
END\ BLOCK # Note backslash. Space will be ignored otherwise
}x;
This does not work. Since the wildcard ('.') matches every character EXCEPT a newline, the regular expression hits a dead end when it gets to the first newline.
Sometimes, as in this case, it is helpful to have the wildcard ('.') match EVERYTHING, not just the newline. And, by extension, to have the wildcard (\s) match [\n\t ], not just tabs and spaces. This is what the modifier 's' does.
In other words it forces Perl to not assume that the string you are working on is one line long. The above then does work with an s on the end of the regular expression:
$line =~ m{
BLOCK(\d+)
(.*?)
END\ BLOCK
}xs;
With the modifier s this now works as expected.
Modifier m is an opposite of the s operator. In other words, it says 'treat the regular expression as multiple lines, rather than one line. This basically makes it so ^ and $; now match not only the beginning and ending of the string (respectively), but also make ^ match any character after a newline, and make $ match a newline. For example,
$line = 'a b c'; $line =~ m"^(.*)$"m;the m modifier will make the backreference $1 become 'a' instead of "a\nb\nc".
Modifier e provides the possibility to evaluate the second part of the s/// as a complete 'mini-Perl program' rather than as a string. This dramatically increases the power of substitution operator in Perl. For example let's assume that you want to substitute all of the letters in the following string with their corresponding ASCII number:
$string = 'hello';
$string =~ s{ ( \w ) } # we save the $1.
{ord($1). " ";} egx;
This example prints out '104 101 108 108 111". Each character was taken in turn here and run through the 'ord' function that turned it into a digit. Needless to say, this can do pretty powerful stuff in a short amount of time. It also runs the risk of being extremely obscure.
We suggest you use this logic as a last resort, when all of your other 'bag of tricks' has failed. Its use can sometimes hide a cleaner way of doing things. Is the above really clear? Or is this better:
$string = turnToAscii($string);
sub turnToAscii{
my ($string) = @_;
my ($return, @letters);
@letters = split(//, $string);
foreach $letter (@letters) {
$letter = ord($letter) . " " if ($letter =~ m"\w");
}
$return = join('', @letters); $return;
}
This latter example is explicit and easily maintainable. However, it is also over 10 times as long and also a few times slower, so a judgment call has to be made on when to use modifier e.
Modifier g in substitution meant that every single instance of a regular expression was replaced. However, this is meaningless in the context of matching. In matching Perl remembers where that match occurs and starts the next matching from this place, not from the beginning of the string. When Perl hits the end of the string, the iterator is reset:
$line = "hello stranger hello friend hello sam";
while ($line =~ m"hello (\w+)"sg){
print "$1\n";
}
This outputs
stranger friend sam
and then quits, because the inherent iterator comes to the end of the expression.
There is one caveat here. With modifier g any modification to the variable being matched via assignment causes this internal iterator to be reset to the beginning for the string.
$word = "hello";
$text=<>;
$i=0;
while ($text =~ m"($word)"sg) {
print "instance $i of the word $word was found\n"
$text="$text\n Word '$word' was found with offset".length($`)."\n";
$i++;
}
As the variable $text is changed inside the loop, the iterator will be reset to the beginning of the string, creating an infinite loop!
This modifier is helpful when you have a complex regex that in inside a nested loop, so the time consumed by matching greatly influence the total time the program runs.
foreach $filesystem (@fstab) {
foreach $file (@files) {
foreach $line in (@text) {
$line =~ m"<complex regular expression>";
}
}
}
By default each time that Perl hits this regular expression, it compiles it. This takes time, and if your regex is complex and does not contain any variable interpolation this is unnecessary operation that can be and should be blocked. While it is not recommended and is a bad style to use modifier o with a regex that contains variable interpolation, Perl allows this. It assumes that you make a promise that after first evaluation the variable that represents regex will never change. If it does, Perl will not notice your change.
perlrequick - perldoc.perl.org
Prev | Up | Contents | Down | Next
Society
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
Quotes
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Bulletin:
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
History:
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
Classic books:
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D
Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|
|
You can use PayPal to to buy a cup of coffee for authors of this site |
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.
Last modified: March 12, 2019
