Re^4: perl process slower and slower when loop number increase

>> Bear in mind that C, Java etc are compiled languages

AFAIK, C is compiled language, Java etc only "compile" to intermediate code, well, AFAIK (again) perl have optimizer that do like java, but no need create intermediate code file like java class

php and nodejs are not "compiled" language, but their loop process not slowing down

>> You haven't shown anything apart from perl taking a time linearly proportional the the number of times the loop is executed

in my vps box:

$ time perl -e ''

real    0m0.003s
user    0m0.000s
sys 0m0.000s

$ time perl -e 'for($i=0;$i<=0;$i++){}'

real    0m0.003s
user    0m0.000s
sys 0m0.000s

$ time perl -e 'for($i=0;$i<=10;$i++){}'

real    0m0.004s
user    0m0.004s
sys 0m0.000s
$ time perl -e 'for($i=0;$i<=100;$i++){}'

real    0m0.004s
user    0m0.000s
sys 0m0.000s
$ time perl -e 'for($i=0;$i<=1000;$i++){}'

real    0m0.003s
user    0m0.000s
sys 0m0.000s
$ time perl -e 'for($i=0;$i<=10000;$i++){}'

real    0m0.005s
user    0m0.004s
sys 0m0.000s
$ time perl -e 'for($i=0;$i<=100000;$i++){}'

real    0m0.015s
user    0m0.012s
sys 0m0.000s
$ time perl -e 'for($i=0;$i<=1000000;$i++){}'

real    0m0.094s
user    0m0.088s
sys 0m0.004s
$ time perl -e 'for($i=0;$i<=10000000;$i++){}'

real    0m0.731s
user    0m0.724s
sys 0m0.004s
$ time perl -e 'for($i=0;$i<=100000000;$i++){}'

real    0m7.253s
user    0m7.256s
sys 0m0.000s
$ time perl -e 'for($i=0;$i<=1000000000;$i++){}'

real    1m13.921s
user    1m13.828s
sys 0m0.008s
[download]

I still think there limit number between $i=100000 and $i=1000000;

but where to set it ?

Comment on Re^4: perl process slower and slower when loop number increase Download Code

Replies are listed 'Best First'.
Re^5: perl process slower and slower when loop number increase by haukex (Archbishop) on Jan 22, 2018 at 16:15 UTC
I still think there limit number between $i=100000 and $i=1000000 What do you mean by this? All you've shown is a fairly linear relationship between the loop count and time, exactly what dave_the_m said. 10000 0.004 100000 0.012 1000000 0.088 10000000 0.724 100000000 7.256 1000000000 73.828	[reply]
Re^5: perl process slower and slower when loop number increase by pryrt (Abbot) on Jan 22, 2018 at 16:22 UTC
I don't know about you, but when I increase a counter's max from n to 10n, I expect the time to go up by 10x. The time for 100_000 is 12-15ms, the time for 1_000_000 is 88-94ms, so that's less than a factor of 10x. If any misinterpretation given your posted data, you should misinterpret that as 1_000_000 behaved better per-iteration than 100_000. (Really, the overhead is dwarfing everything else at that few iterations.) 1e6 to 1e7 shows <10x increase, so that's still okay; 1e7 to 1e8 shows almost exactly 10x increase, so that's right. 1e8 to 1e9 shows only 2x increase: this is the first where I'd think you were hitting a limit of some sort. What's the output of your `perl -V \| grep size`, especially `perl -V:ivsize`? Even with 4, I would have expected the 1e9 loop to increase linearly, because 1e9 only requires 30bits, which easily fits inside a 32-bit signed IV at 4-bytes. On my super-ancient system with a 4byte ivsize, `#/usr/bin/perl use warnings; use strict; use Benchmark qw(:hireswallclock timethese cmpthese); my $r = timethese( -60 , { oneE5 => sub { for( 1 .. 1e5 ) {} }, oneE6 => sub { for( 1 .. 1e6 ) {} }, oneE7 => sub { for( 1 .. 1e7 ) {} }, oneE8 => sub { for( 1 .. 1e8 ) {} }, oneE9 => sub { for( 1 .. 1e9 ) {} }, }); cmpthese($r);` [download] gives me oneE5: 63.876 wallclock secs (63.43 usr + 0.20 sys = 63.63 CPU) +@ 140.81/s (n=8960) oneE6: 63.7849 wallclock secs (63.09 usr + 0.23 sys = 63.32 CPU) + @ 14.17/s (n=897) oneE7: 63.6703 wallclock secs (63.05 usr + 0.21 sys = 63.26 CPU) + @ 1.42/s (n=90) oneE8: 62.9542 wallclock secs (62.30 usr + 0.20 sys = 62.50 CPU) + @ 0.14/s (n=9) oneE9: 74.5811 wallclock secs (73.81 usr + 0.26 sys = 74.07 CPU) + @ 0.01/s (n=1) (warning: too few iterations for a reliable count) Rate oneE9 oneE8 oneE7 oneE6 oneE5 oneE9 1.35e-02/s -- -91% -99% -100% -100% oneE8 0.144/s 967% -- -90% -99% -100% oneE7 1.42/s 10438% 888% -- -90% -99% oneE6 14.2/s 104829% 9738% 896% -- -90% oneE5 141/s 1042910% 97688% 9798% 894% -- [download] which looks highly linear to me, through the 1-billion (1e9). I don't know why your 1e9 is only 2x your 1e8, because I cannot repeat that. But my guess, as hinted above, is ivsize. Or maybe it's the c-ish loop that hippo pointed out; you could test that by also including a couple more rows in the cmpthese() with c-style loops for 1e8 and 1e9. Also, longer iterations (like 600sec instead of 60sec) will give more accurate results; this was just a proof of concept. update: after reading haukex's reply, and wondering where the 73.8s came from, I re-read your timing, and saw the "1m" prefix on the output.. I was only seeing the "13.828s", which confused me. Now that I see 1m13.828s, that's ~74s, which as haukex said, is reasonably linear. So my confusion is allayed, so my requests for ivsize info is unnecessary.	[reply] [d/l] [select]