Re^2: counting words in string

Since I'm a not an expert with perl regex, I start digging in the code of haukex with commenting it's original code and with searching a simpler loop.

Here we go:

use warnings;
use strict;
use Test::More tests=>2;

my $str = "iowq john stepy andy anne alic bert stepy anne bert andy st
+ep alic andy";

my %names;
=for comment
pos 
Returns the offset of where the last m//g search left off for the vari
+able in question ($_ is used when the variable is not specified). 
Note that 0 is a valid match offset. 
undef indicates that the search position is reset (usually due to matc
+h failure, but can also be because no match has yet been run on the s
+calar).
=cut

pos($str)=undef;

=for comment
https://www.regular-expressions.info/continue.html
The position where the last match ended is a "magical" value that is r
+emembered separately for each string variable. 
The position is not associated with any regular expression. 
This means that you can use \G to make a regex continue in a subject s
+tring where another regex left off.
If a match attempt fails, the stored position for \G is reset to the s
+tart of the string. To avoid this, specify the continuation modifier 
+/c.
=cut

while ($str=~/\G #start where the last match ended
    \s*  #match 0 to n space char
    (\S+) #remember any non space char after that and followed by
    (?:  #start clustering of
    \s+|\z #1 to n spaces or the end of the string
    ) #end clustering
    /gcx) {
    $names{$1}++;
}
die "failed to parse \$str" unless pos($str)==length($str);

test_it (\%names);

%names = ();
#Takes a new variable
#my $str2 = "iowq john stepy andy anne alic bert stepy anne bert andy 
+step alic andy";
#or reset pos for the original var
pos($str)=undef;
my $last;
while ($str=~/(\w+)/g) {
    #print $1, " ", pos $str, "\n";
    $names{$1}++;
    $last = pos $str;
}

die "failed to parse \$str" unless $last ==length($str);

test_it(\%names);

sub test_it {
    my $hr_names = shift;
    is_deeply $hr_names, { alic => 2, andy => 3, anne => 2, bert => 2,
    iowq => 1, john => 1, step => 1, stepy => 2 };
}
[download]

I have 3 questions

Where is the /c modifier documented in the perldoc ? It is given at the end of pos but I couldn't find no other description
How comes that I have to remember pos $str in the second loop ? It's undef after the second loop but not after the first one
Are the two loops equivalent or will the second one failed in some situation ?

Cheers

François

Comment on Re^2: counting words in string Download Code

Replies are listed 'Best First'.
Re^3: counting words in string by haukex (Archbishop) on Aug 10, 2018 at 21:44 UTC
`pos($str)=undef;` Note: This may seem like overkill in the simple example I showed, but it might become important if the code is copy and pasted into a larger script, where it's possible that a previous regex operation on `$str` left its pos set (but we want to start matching at the beginning of the string). Where is the /c modifier documented in the perldoc ? Hm, you're right that the doc seems a little hard to find. There's a pretty good explanation in perlretut's "Global Matching", and a more complex example of the usage of `m/\G.../gc` in perlop under "`\G` assertion". How comes that I have to remember pos $str in the second loop ? That would be the effect of the `/c` modifier: your second loop ends when the regex fails to match. The regex failing to match also resets pos, except when `/c` is used. Are the two loops equivalent or will the second one failed in some situation ? The way you've written it, no, because `\w+` is not the same as `\S+`: the latter will match any non-word characters too, anything except `\s` characters. If I change your loop's regex to `/(\S+)/g`, you can get the "`failed to parse $str`" error, namely when there's whitespace at the end of `$str`. This is because the last match doesn't reach all the way to the end of the string, while the regex I showed does, because it includes a `\s+` after the `\S`. The `pos==length` check makes sense if you want to make sure your loop went all the way to the end of the string and there's no garbage left at the end that we didn't match, but in this case, the only thing that could possibly be left after matching `\S+` is whitespace, which we're not interested in. searching a simpler loop I showed the `m/\G.../gc` technique because I wanted to demonstrate it - it's a great tool for certain types of parsing (one example in the first paragraph of this post, or this), but as long as the OP just wants to split on whitespace, it's admittedly overkill. If you wanted to simplify the loop, you could say this (others have shown variations of this): `my %names; while ($str=~/(\S+)/g) { $names{$1}++; }` [download] But as soon as the matching rule isn't as simple as `\S+`, things can get tricky. For example, if the rule is `\w+`, and the input string is `" foo bar "`, anything that looks for `\w+` only* will miss the junk at the end of the string.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^3: counting words in string
by haukex (Archbishop) on Aug 10, 2018 at 21:44 UTC

pos($str)=undef;

Note: This may seem like overkill in the simple example I showed, but it might become important if the code is copy and pasted into a larger script, where it's possible that a previous regex operation on $str left its pos set (but we want to start matching at the beginning of the string).

Where is the /c modifier documented in the perldoc ?

Hm, you're right that the doc seems a little hard to find. There's a pretty good explanation in perlretut's "Global Matching", and a more complex example of the usage of m/\G.../gc in perlop under "\G assertion".

How comes that I have to remember pos $str in the second loop ?

That would be the effect of the /c modifier: your second loop ends when the regex fails to match. The regex failing to match also resets pos, except when /c is used.

Are the two loops equivalent or will the second one failed in some situation ?

The way you've written it, no, because \w+ is not the same as \S+: the latter will match any non-word characters too, anything except \s characters.

If I change your loop's regex to /(\S+)/g, you can get the "failed to parse $str" error, namely when there's whitespace at the end of $str. This is because the last match doesn't reach all the way to the end of the string, while the regex I showed does, because it includes a \s+ after the \S.

The pos==length check makes sense if you want to make sure your loop went all the way to the end of the string and there's no garbage left at the end that we didn't match, but in this case, the only thing that could possibly be left after matching \S+ is whitespace, which we're not interested in.

searching a simpler loop

I showed the m/\G.../gc technique because I wanted to demonstrate it - it's a great tool for certain types of parsing (one example in the first paragraph of this post, or this), but as long as the OP just wants to split on whitespace, it's admittedly overkill. If you wanted to simplify the loop, you could say this (others have shown variations of this):

my %names;
while ($str=~/(\S+)/g) {
    $names{$1}++;
}
[download]

But as soon as the matching rule isn't as simple as \S+, things can get tricky. For example, if the rule is \w+, and the input string is " foo bar *", anything that looks for \w+ only will miss the junk at the end of the string.

[reply]
[d/l]
[select]