slow perl script interpretation (syntax check)

Robot has asked for the wisdom of the Perl Monks concerning the following question:

Hi!

Recently, I was shocked to discover that interpreting large perl file was significantly slower on my newer machines (they run fedora core 5+ and centos 5) than my older machines (they run fedora core 2 and core 3).

I am checking the times using "time perl -c <file>" and while the older OSes were doing this for ~1-2 secs (it's a huge CGI script) the newer OSes were parsing it for more than 10 seconds.

The problem is neither in perl version as I checked with both versions I use (5.8.5 and 5.8.8), nor in hardware as I checked with different machines (even vmware ones).

I've tried to do a timed strace but the only thing I was able to discover is that times are generally better on the newer OSes but times between memory segment changes (brk(0x.....) command) were actually causing the problem:

...
brk(0xaebb000)                          = 0xaebb000 <0.000013>
brk(0xaee4000)                          = 0xaee4000 <0.000018>
brk(0xaf0d000)                          = 0xaf0d000 <0.000018>
brk(0xaf03000)                          = 0xaf03000 <0.000023>
brk(0xaf2c000)                          = 0xaf2c000 <0.000023>
...
[download]

as you can see the system times are fast but the user time are very slow (~ 8-9 seconds waiting till these brk's appear one by one)

Could anyone give me a clue what might be the root of the problems or more hints and ideas of research?

Thank you for you time and efforts!

------ UPDATE (SOLUTION) ------

First, I want to thank to all of the guys here that gave me valuable ideas and tips.

The problem seems to hide in newer glibc working with i386-compiled perl (that's only a humble guess, I would gladly accept any other truthful explanation). I downloaded a new Fedora SRPMS, rebuilt it with i686 target and forcibly installed the newly built RPM over the existing one. Nevertheless that "perl -v" still shows that it's i386 version you should not believe it. Now perl seems to be fast enough (will need a couple more tests but it seems real now)! :)

Comment on slow perl script interpretation (syntax check) Download Code

Replies are listed 'Best First'.
Re: slow perl script interpretation (syntax check) by roboticus (Chancellor) on Oct 11, 2007 at 12:41 UTC
Robot: Check differences in your path--perhaps you have a slow network drive in your @INC path that perl has to look through? You may also want to check your I/O stats to see if you're thrashing the machine somehow. That's all that comes to mind right now. ..roboticus	[reply]
Re^2: slow perl script interpretation (syntax check) by Robot (Novice) on Oct 11, 2007 at 13:07 UTC
Thank you very much for your proposals. There are no additional @INC paths (actually I can see their traversal with strace and it's instant) and I am also testing on both vmware and real machines. About the I/O stats - the only thing I noticed is that because of that behaviour one of the CPUs on the "problematic new OSes" always get burnt up to 100%. Additional info: The older OSes are behaving just like the perl file was bytecoded. Actually if I bytecode it, it starts to check syntax for the same amount of seconds on the new OSes. Unfortunately, bytecoding is not a production-stable variant to use ...	[reply]
Re: slow perl script interpretation (syntax check) by dave_the_m (Monsignor) on Oct 11, 2007 at 22:50 UTC
I seem to recall that some versions of Fedora supplied a perl binary with internal debugging enabled, which may slow things down a bit. You can see whether this is the case as follows: $ perl -Dt -e 1 Recompile perl with -DDEBUGGING to use -D switch (did you mean -d ?) $ [download] In this case, it hasn't got debugging compiled in. Dave.	[reply] [d/l]
Re^2: slow perl script interpretation (syntax check) by Robot (Novice) on Oct 13, 2007 at 21:13 UTC
10x, Dave! Unfortunately, that's not the case (just checked all of them - they haven't got debugging compiled in).	[reply]
Re: slow perl script interpretation (syntax check) by perlfan (Parson) on Oct 11, 2007 at 18:05 UTC
You may want to use something like Devel::Trace or `perl -d:DProf` to see if you can spot where things are slow in the code. Running memcheck and some io bechmark on the system itself might find something, too....in otherwords, you need to dig a lot deeper.	[reply] [d/l]
Re^2: slow perl script interpretation (syntax check) by Robot (Novice) on Oct 13, 2007 at 21:24 UTC
Yes, we even use an "enhanced" trace and debug package of our own but it doesn't seem like a perl error, i.e. all is executing normally when traced as a perl but it's the initial interpretaion/syntax check that's slowing the things on the "new" OSes. I will try to do some io testing but I will be suprised if that's the problem as it happens on different hardware platforms and OSes (FC5, CentOS5 tested for now). Thank you! P.S. I wonder ... could it be glibc issue, as all of the FC5+ are using glibc-2.4 while the older use glibc-2.3	[reply]
Re: slow perl script interpretation (syntax check) by Robot (Novice) on Oct 16, 2007 at 14:43 UTC
------ UPDATE (SOLUTION) ------ First, I want to thank to all of the guys here that gave me valuable ideas and tips. The problem seems to hide in newer glibc working with i386-compiled perl (that's only a humble guess, I would gladly accept any other truthful explanation). I downloaded a new Fedora SRPMS, rebuilt it with i686 target and forcibly installed the newly built RPM over the existing one. Nevertheless that "perl -v" still shows that it's i386 version you should not believe it. Now perl seems to be fast enough Now perl seems to be fast enough (will need a couple more tests but it seems real now)! :)	[reply]