The conclusion is that all regular expressions written like this: $text =~ /(.*?)<whatever>/ take a thousand times more on 5.8.0. The same expressions written as $text =~ /^(.*?)<whatever>/ which obviously means the same thing (look for the first occurence of <whatever> and save the text preceding it in the corresponding variables) has the same performance implications across these two versions.

There is no "easy" way to fix this problem in the source of perl; you have to explicitly state the anchor yourself. The reason is that perl has no way of knowing whether or not you'll end up using what you captured as a backreference, so anchoring has an unknown effect. The problem is not only when the .* is captured, either; any capturing in the regex causes a problem.

(The case of "abc\ndef1" =~ /.*\d/ is already handled by the engine so as not to fail. It would fail if the regex were treated as /^.*\d/, but the engine makes it (?m:^) if necessary.)

Thank you for your quick answer, and thanks again for taking the time to explain. Although I agree that the two regular expressions have different meanings, the real question here is why Perl 5.6.1 is 500-1,000 times faster than Perl 5.8.0 on the same regular expression -- this is my real query. Am I to assume that Perl 5.6.1 did not properly parse certain regular expressions and Perl 5.8.0 now does? I just tried your regular expressions and they yielded the same results under both versions. How unstable is my previous code, if new versions can make it obsolete in performance, as if encouraging not to upgrade.

#reg.pl
$s = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxyyyRRRRyyyy\n" x 500;
$n = 0;
$n++ while ($s =~ /(.*?)RRRR\1/sg);
print "$n matches\n";
[download]

time ~/bin/perl5.8.0 reg.pl 
500 matches

real    0m4.836s
user    0m4.800s
sys     0m0.010s

time ~/bin/perl5.6.1 reg.pl 
0 matches

real    0m0.020s
user    0m0.020s
sys     0m0.000s

So, in fact, you are complaining that a bug got fixed. The problem is that these are extremely inefficient regular expressions because they involve a lot of backtracking. I recommend reading Mastering Regular Expressions for a detailed explanation.

I'm not entirely sure why the regexes were so much slower, unless they just never could actually match. In that circumstance, /.*FAIL/ would be a lot slower than /^.*FAIL/.
_____________________________________________________
Jeff japhy Pinyan, P.L., P.M., P.O.D, X.S.: Perl, regex, and perl hacker
How can we ever be the sold short or the cheated, we who for every service have long ago been overpaid? ~~ Meister Eckhart

Ok so japhy told you why that is slow, here's a way to make your code fast regardless - don't even bother with the capturing.

Capturing is always slow because it has to make a copy of the source string. $1, internally, is just substr( $safe_copy_of_match, $-[1], $+[1] - $-[1] ). So the largest speed hit (that I'm aware of) is the memory operation of making a safe duplicate of the data that was just matched. COW (copy on write) may mitigate this if/when it ever gets into perl.

Likely to be be fastest. This was my second thought.

my $whatever_index = index lc $text , $whatever;
return( substr( $text, 0, $whatever_index ),
        substr( $text, $whatever_index + length $whatever ) );
[download]

This may be the fastest. It was my third thought.

my $whatever_index = index lc $text, $whatever' ;
my $whatever_length = length $whatever;
return unpack "a" . $whatever_index . "x" . $whatever_length . "a*", $
+text;
[download]

This was my first thought. Use a plain regex to *locate* the thing in the string and then just substr() the equivalent of the captures out. This happens to be simplest to look at so it wins on the visual-complexity scale. This is a great general technique to avoid capturing on regexes and as such is a great post-bechmarking optimization.

if ( $text =~ /whatever/i )
{
    return( substr( $text, 0, $-[0] ),
        substr( $text, $+[0] );
}
[download]

Perl 5.8.0 is slow on your system because Red Hat compiled it with threads and debugging turned on (which you didn't do in your 5.8.5 compile) and because they set the locale to use unicode and folded in a bunch of patches for unicode that were not in the official 5.8.0 release. This has been written about extensively. See the Red Hat bugzilla for more details. This was fixed in 5.8.1. The remaining slowdown of 3.7 is probably due to the regex change that japhy mentioned.

I can tell you one thing: if IBM had written Perl, this would have never happened. Maybe there aren't enough alpha and beta testers, maybe developers don't have the time to write enough warning messages. What's certain is that Perl is not seen as a product, and the members of the community it attempts to serve are not being looked upon as customers. And that's the very difference between Open source and closed source software. What good is it's free, if it is deceiving its users about the problems it claims to solve?

Do you often find that insinuating that people are ignorant, malicious, sloppy, or stupid makes them likely to help you?

Maybe I was trying to convey too many ideas and feelings out of context (on one hand I'm carefully explaining the problem and seeking professional advice, on the other hand I'm criticizing the ones responsible). In the right context, anything can be made to sound the way it was intended, and I do apologize if this particular bit sounded too harsh.

I understand the frustration. It's sometimes difficult to remember that dozens of people have put thousands of hours into a project given away freely for other people to use when you find an apparent bug, but it's very wise to keep that in mind.

Your description of the problem was very good, though.

I think the point is that you don't need to worry about that if you're talking to a commercial vendor. It's a shift in attitude that anyone moving to F/OSS, hopefully, will get used to.

I've talked to proprietary vendors before. Maybe some don't cause you to worry, but those I can think of did not inspire me with confidence.

Barnraising your IT might be an interesting read.

Make sure to read the comments on sentiments about commercial vendors and contracts.

Makeshifts last the longest.

Not sure I agree with *that*... As an old mainframe programmer, I can tell you that when COBOL went from COBOL to VS COBOL II back in the late 80s, we practically had to recompile our entire mainframe library.

And it wasn't like it was obscure stuff... They did away with the EXAMINE statement, which was a staple of COBOL development.

It did away with the ON statement... and would no longer accept LABEL RECORDS...

Worst of all, the TRANSFORM statement vanished.

They had (supposedly) good reasons for making those kind of fundamental changes, but it didn't change the fact that COBOL, arguably the de facto programming standard of the time, was fundamentally changed long after it was a mature product.

So, don't be so sure that IBM wouldn't have done the same thing... they've done it before :)

Trek

open (FILE, "a.txt");
$/=undef;
$txt = <FILE>;
$txt =~ s/whatever//sig;
[download]

Finding the things around 'whatever' is different than just removing 'whatever'.

My code was just a literal translation of the OP's remark...

"As you can see, this code slurps a file and removes all occurences of a certain word (`whatever')."

---

-- All code is 100% tested and functional unless otherwise noted.

Take a look at how that extract() routine is used a little more closely...

I was trying to make a point about the fact that this code runs incredibly slower on the prepackaged RH9 Perl 5.8.0 compared to the prepackaged (Mandrake 8 I believe) Perl 5.6.1. The example above was whipped up especially for this experiment, after a period of tracking down the exact pieces of code which were slowing down my original Perl programs.

Only after noticing that the =~ /(.*?) constructs were leading to neverending pauses in the Perl 5.8.0 code, did I realize that adding a ^ anchor would eliminate the inherent ambiguity (the /s switch was on). That's how I made this short example, in which I added the extract subroutine so I can get clear results in the DProf debugger and can make direct comparisons against the Perl versions. I was astounded to see that the slow ratio was not within 1.0 and 2.0 (meaning a tad slower), but somewhere between 500.0 and 1,000.0, explaining why buying new hardware was definitely more expensive than having somebody replace all /(.*?) regexps to /^(.*?) :).

This is an issue of good Perl and bad Perl

You are correct. .*? is perhaps one of the least efficient singular regex constructs available. Why are you matching text you are not keeping, anyways? Are you unaware that there is an entirely separate construct (s/whatever//) made for removing text?

Have you not read the extensive perlre documentation for the product that you are using? Just because something is free doesn't mean that you automatically know how to use it right-out-of-the-box.

Also, if IBM had written Perl, it would probably take over a minute to start while it loaded its built in WSADIE plugins for J2EE development.

I am going to guess that you have a UTF-8 LANG set in your environment.

A consequence of this in 5.8.0, but not in 5.6.1 or 5.8.5, is that your file is implicitly opened as UTF-8. This may seem minor, but because you included the /i modifier, it probably slowed it down a lot, since case insensitivity in Unicode is a lot more complicated. You could test this by modifying your environment and rerunning, or by explicitly opening the file as latin-1, or by removing the /i.

You seem to discount the speed up you saw between 5.6.1 and 5.8.5 with your second regex version. I don't think this is really fair. I suspect that the regex engine really is faster in the later versions, when they are actually doing the same thing.

The problem really seems to be that due to some subtleties in how certain things work in different versions of perl, the regex engine is not doing the same things in each of your cases. Since you are so willing to criticize the Perl community, I will gladly turn around and criticize you. This is not particularly obscure information. It's pretty well explained in perldelta, perlunicode, and other man pages. You apparently made the decision to upgrade perl versions without taking the time to research what changed. 5.6 to 5.8 is not a minor change: there are significant changes between the two which you would have been well advised to consider before making the switch.

Furthermore, did you even stop to wonder why there were additional functions being called in one case and not the others? Don't you think this ought to have been a clue that things were not as simple as you would like to think?

I am content about the one thing that I find relevant: my Perl codebase was successfully ported to Perl 5.8.5. My blaming the Perl community and other people's blaming me have both proved besides the point and their only merit -- artistic at most. As seems to always be the case, those people who had only techical points to make were the most useful.