Re^3: Get a known substring from a string

I know it looks really trivial once you see it, but I'm really astonished by your approach of using 1+index(...) - it had not occurred to me to use index that way in an expression to check for presence. I'll add that to my set of idiosyncratic phrases, just like if( system(...) == 0 ) { for successful execution of subprocesses.

Update: I wondered about how much the capturing parentheses cost, and it seems they account for roughly ~~a third~~ half of the performance attainable when using the regex engine. Maybe the two additional steps executed in the regex engine (OPEN1 and CLOSE1) are to blame for that, as they effectively double the number of steps the regex engine has to execute for a successful match.

Not invoking the regex engine still is much faster, even though I had thought there once was an optimization that turned constant regular expressions without anchors or quantifiers into an index lookup...

# a:  if( $s =~ m[(lazy)] ){ $found=$1 }
Compiling REx "(lazy)"
Final program:
   1: OPEN1 (3)
   3:   EXACT <lazy> (5)
   5: CLOSE1 (7)
   7: END (0)
anchored "lazy" at 0 (checking anchored) minlen 4
Matching REx "(lazy)" against "the quick brown fox jumps over the lazy
+ dog"
Intuit: trying to determine minimum start position...
  Found anchored substr "lazy" at offset 35...
  (multiline anchor test skipped)
  try at offset...
Intuit: Successfully guessed: match at offset 35
  35 < the > <lazy dog>      |  1:OPEN1(3)
  35 < the > <lazy dog>      |  3:EXACT <lazy>(5)
  39 <the lazy> < dog>       |  5:CLOSE1(7)
  39 <the lazy> < dog>       |  7:END(0)
Match successful!
Freeing REx: "(lazy)"
# b:  $found = 'lazy' if 1+index( $s, 'lazy' );
# c:  if( $s =~ m[lazy] ){ $found=$& }
Compiling REx "lazy"
Final program:
   1: EXACT <lazy> (3)
   3: END (0)
anchored "lazy" at 0 (checking anchored isall) minlen 4
Matching REx "lazy" against "the quick brown fox jumps over the lazy d
+og"
Intuit: trying to determine minimum start position...
  Found anchored substr "lazy" at offset 35...
  (multiline anchor test skipped)
  try at offset...
Intuit: Successfully guessed: match at offset 35
Freeing REx: "lazy"
       Rate    a    c    b
a 2038631/s   -- -50% -75%
c 4089154/s 101%   -- -49%
b 8013601/s 293%  96%   --
[download]

The program I used:

use strict;
use Benchmark 'cmpthese';
use vars '$s';
$s='the quick brown fox jumps over the lazy dog'; 
my $found;

my %benchmarks = (
    a => q[ if( $s =~ m[(lazy)] ){ $found=$1 } ],
    b => q[ $found = 'lazy' if 1+index( $s, 'lazy' ); ],
    c => q[ if( $s =~ m[lazy] ){ $found=$& } ],
);

{
    use re 'debug';
    for (sort keys %benchmarks) {
        print "# $_: $benchmarks{$_}\n";
        undef $found;
        my $code = eval qq{sub { $benchmarks{$_} } }
            or die "Couldn't compile benchmark $_: $@";
        $code->();
        $found eq 'lazy'
            or die "Unexpected results: [$found] vs. 'lazy'";
    };
};

cmpthese( -1, \%benchmarks);
[download]

Comment on Re^3: Get a known substring from a string Select or Download Code

Replies are listed 'Best First'.
Re^4: Get a known substring from a string by BrowserUk (Patriarch) on Sep 10, 2016 at 11:40 UTC
I'm really astonished by your approach of using 1+index(...) - it had not occurred to me to use index that way in an expression to check for presence. I've been using it that way for as long as I can remember. This turns up 50+ of my uses here with the earliest being in September 2002 which is only a few months after I started coming here. Adding the one has another benefit when doing searches in a loop: `##do something with $p - 1 while $p = 1 + index( $haystack, $needle, $ +p );` [download] That of automatically moving the start point along after each match. You have to remember to subtract 1 when using the match point; but that's no more onerous than remembering to increment it. I had thought there once was an optimization that turned constant regular expressions without anchors or quantifiers into an index lookup... That optimisation is there, and can be even quicker than index but you have to code it exactly correctly for it to kick in:: `$s = 'the quick brown fox jumps over the lazy dog'; cmpthese -1,{ a => q[ if( $s =~ m[(lazy)]){ $found=$1 } ], b => q[ $found = 'lazy' if 1+index( $s, 'lazy' ); ], c => q[ $found = 'lazy' if $s =~ 'lazy'; ], };; Rate a b c a 577066/s -- -77% -79% b 2492720/s 332% -- -11% c 2791311/s 384% 12% -- [0]{} Perl>` [download] Unfortunately, it doesn't generalise. Even using a variable instead of literal means some of that performance gain is lost: $s = 'the quick brown fox jumps over the lazy dog'; $x = 'lazy'; cmpthese -1,{ a => q[ if( $s =~ m[($x)]){ $found = $1 } ], b => q[ $found = $x if 1 + index( $s, $x ); ], c => q[ $found = $x if $s =~ $x ], };; Rate a c b a 449697/s -- -79% -82% c 2167332/s 382% -- -12% b 2462877/s 448% 14% -- $s = 'the quick brown fox jumps over the lazy dog'; $x = 'lazy'; cmpthese -1,{ a => q[ if( $s =~ m[($x)]){ $found = $1 } ], b => q[ $found = $x if 1 + index( $s, $x ); ], c => q[ $found = $x if $s =~ $x ], };; Rate a c b a 459542/s -- -79% -80% c 2184810/s 375% -- -6% b 2318112/s 404% 6% -- [download] The nature of the type of work I do means that I learnt early on to only start the regex engine if I needed regex. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :) In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l] [select]
Re^5: Get a known substring from a string by flowdy (Scribe) on Sep 13, 2016 at 07:34 UTC
Don't get me wrong - TIMTOWTDI. But how is that expression nearly as clear as `index($str, $whatever) > -1`? For the system, it is probably equivalent regarding the overall number of operations. For the human brain (mine at least), it is one less. Select wisely the idioms you use. Others reading your code will thank you. I, for one, would not use that one because I find it is just for the sake of brevity without its being worth one more thought every time it is read. `while ( ($p = index $haystack, $needle, $p+1) > -1 ) { # e.g. (not tested) pos($haystack) = $p; $p = pos($haystack) if $str =~ s{ \G ... }{...}cxms # regex doing the tough dishes ; # /c flag: preserve pos() on non-match }` [download]	[reply] [d/l] [select]
Re^6: Get a known substring from a string by BrowserUk (Patriarch) on Sep 13, 2016 at 12:05 UTC
Select wisely the idioms you use. Believe me, I do! I, for one, would not use that one because I find it is just for the sake of brevity without its being worth one more thought every time it is read. So you optimise your code in order to save the maintenance programmer, "one more thought", during each of his occasional visits over the future life of the code. Say that "one more thought" requires 5 seconds; or 10, and the code stays in use for 10 years and the maintenance programmer visits it one every 6 months, or three; over the lifetime of the code you've "saved" a maximum of 400 seconds (0.11 hours) of programmer time. Now. let's say that the difference in performance of that one usage, is 1% on the overall runtime; and over that 10 year lifetime the program runs for 8 hours a day every work day for those 10 years, then you have cost the users of that program 208 hours of their time. The owners 208 hours of extra processor wear, electricity and air conditioning costs. Now lets say there are 3 uses of that idiom; and half a dozen uses each of half a dozen other efficient idioms that you reject; and the difference in runtime isn't just 1% but 25%. Now your "potential" savings in programmer time over 10 years becomes 1236 seconds, A whole 20 minutes! But your cost to the users and owners is: 5000 hours! Of user time, cpu resource, energy. Now, perhaps you are a webcoder and the entire idea of code running for 8 hours per working day is an anathema; so think of it in terms of potential customers lost because your maintenance programmer friendly code causes the page load time to move from 2.75 seconds to 3.25 seconds and your impatient target audience of millennial teens are known to move on if a load takes more than 3 seconds. How many "one more thoughts" do you need to save the maintenance programmers during their occasional visits over the next 10 years to make up for hundreds or thousands or tens of thousands of incomplete loads per day of your middling-sized 365/24/7 commercial website? Now consider that much of the code I write runs on tens (and sometimes thousands) of cores concurrently, and for days and weeks at a time. 1% saving can save my clients thousands of dollars. On at least one occasion hundreds of thousands; per run. And it is not unusual for my code and algorithms to save not 1% or 10% of runtime; but frequently 50% or 80% and on occasion 96%. Ie. runtimes projected to be multiple years, now complete in weeks or even days. And these are not one-offs, but regular gigs over the last 15 years -- I've made a pretty good living at it. And beyond the occasional novel algorithm, a large part of that work has been underpinned by writing what is often fairly prosaic algorithms as efficiently as Perl allows. Ie. Using Perl to its best effect. And finally, everyone who has read my post and your reply -- including you -- will, barring a catastrophically short memory, now always instantly recognise the idiom we are discussing and almost subconsciously recognise its equivalence to your preferred form. That is the very purpose of, nay even the definition of, "idioms". They are 'patterns of usage'; short forms of more complex code that, with regular usage and familiarity become as recognisable as the languages native constructs. They become as a part of the language to those who have taken the time to become familiar with them. If your Perl code is used as a training aid -- you're a teacher, lecturer, or book writer -- then your "optimise code for the casual reader, newbies and maintenance programmers" stance has some merit; otherwise it is the very definition of premature (naive, unfounded, illogical and thoughtless) pessimisation. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :) In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^7: Get a known substring from a string by flowdy (Scribe) on Sep 14, 2016 at 19:03 UTC
Re^8: Get a known substring from a string by BrowserUk (Patriarch) on Sep 14, 2016 at 20:16 UTC
Re^6: Get a known substring from a string by Anonymous Monk on Sep 13, 2016 at 09:16 UTC
Hi I'm using your node as inspiration , to grow some food for fthought :) But how is that expression nearly as clear as ... How do you judge clarity? Its like the whole `if( not $this) .... if(!$this)...` versus `unless( $this )`... yeah to some people one is more natural and the other is confusing some ~~admonish~~critic others for using unless But then , you can say that about every piece of syntax in programming if its your first day Often you see beginners baffled by various syntax, they call it weird, or unusual ... and then it turns out they never read the documentation for the syntax, its as if this thing they havent studied is something new and unknowns, how bizzare that they dont magically know it just by becoming aware of its existence :D I dont know, my first month programming perl I have found using "unless" a bit weird, but then I also found "if not" weird -- weird, its just means new By second month, its all just another piece of syntax, another verb in the vocabulary, second grade grammar :) I run into code combinations I haven't practiced or so even seen , which I don't understand, all the time, it doesn't looks wrong at a glace to maintenance programmer, so it takes a minute or more to decipher the code, thats ok, it wasn't instant to write it gives you some Untillian Headache, thats ok too, you'll learn, it will pass Programming is much more than "just writing code"., Programming is one of the most complex endeavours man undertakes...its ludicrous to believe anyone who might be reading unfamiliar code should be able to instantly and correctly understand another's programs if you asked a brain surgeon to limit himself to those procedures that could be performed by an intern, or the eye surgeon to only use stitches that could be removed/replicated by a staff nurse, then medicine would be in a pretty poor state. Knowing your audience is a good idea to keep in mind, but Production source code in not the place to educate new programmers; nor document design decisions; nor record board meeting deliberations. :) keep programming fun, keep learning and trying new things, repeat, repeat, repeat :)	[reply] [d/l] [select]

Now, perhaps you are a webcoder

How many "one more thoughts" do you need

Now consider that much of the code I write runs on tens (and sometimes thousands) of cores concurrently,

And finally,