Re^2: perl process slower and slower when loop number increase

Replies are listed 'Best First'.
Re^3: perl process slower and slower when loop number increase by dave_the_m (Monsignor) on Jan 22, 2018 at 14:54 UTC
Bear in mind that C, Java etc are compiled languages. The compiler can take as long as it likes to optimise code, and is likely to detect an empty loop and optimise it away. Perl does its compilation as part of its execution, and can therefore only spend a limited amount of time applying optimisations. In particular, it won't optimise away a for(;;) loop with an empty body. So the timings you see for perl are a fixed compilation overhead, followed by N executions of a sequence of opcodes which collectively increment $i, test it against a limiting value, and jump back. You haven't shown anything apart from perl taking a time linearly proportional the the number of times the loop is executed. It's not clear to me why you find that surprising. Dave.	[reply] [d/l]
Re^4: perl process slower and slower when loop number increase by Anonymous Monk on Jan 22, 2018 at 15:43 UTC
>> Bear in mind that C, Java etc are compiled languages AFAIK, C is compiled language, Java etc only "compile" to intermediate code, well, AFAIK (again) perl have optimizer that do like java, but no need create intermediate code file like java class php and nodejs are not "compiled" language, but their loop process not slowing down >> You haven't shown anything apart from perl taking a time linearly proportional the the number of times the loop is executed in my vps box: $ time perl -e '' real 0m0.003s user 0m0.000s sys 0m0.000s $ time perl -e 'for($i=0;$i<=0;$i++){}' real 0m0.003s user 0m0.000s sys 0m0.000s $ time perl -e 'for($i=0;$i<=10;$i++){}' real 0m0.004s user 0m0.004s sys 0m0.000s $ time perl -e 'for($i=0;$i<=100;$i++){}' real 0m0.004s user 0m0.000s sys 0m0.000s $ time perl -e 'for($i=0;$i<=1000;$i++){}' real 0m0.003s user 0m0.000s sys 0m0.000s $ time perl -e 'for($i=0;$i<=10000;$i++){}' real 0m0.005s user 0m0.004s sys 0m0.000s $ time perl -e 'for($i=0;$i<=100000;$i++){}' real 0m0.015s user 0m0.012s sys 0m0.000s $ time perl -e 'for($i=0;$i<=1000000;$i++){}' real 0m0.094s user 0m0.088s sys 0m0.004s $ time perl -e 'for($i=0;$i<=10000000;$i++){}' real 0m0.731s user 0m0.724s sys 0m0.004s $ time perl -e 'for($i=0;$i<=100000000;$i++){}' real 0m7.253s user 0m7.256s sys 0m0.000s $ time perl -e 'for($i=0;$i<=1000000000;$i++){}' real 1m13.921s user 1m13.828s sys 0m0.008s [download] I still think there limit number between $i=100000 and $i=1000000; but where to set it ?	[reply] [d/l]
Re^5: perl process slower and slower when loop number increase by haukex (Archbishop) on Jan 22, 2018 at 16:15 UTC
I still think there limit number between $i=100000 and $i=1000000 What do you mean by this? All you've shown is a fairly linear relationship between the loop count and time, exactly what dave_the_m said. 10000 0.004 100000 0.012 1000000 0.088 10000000 0.724 100000000 7.256 1000000000 73.828	[reply]
Re^5: perl process slower and slower when loop number increase by pryrt (Abbot) on Jan 22, 2018 at 16:22 UTC
I don't know about you, but when I increase a counter's max from n to 10n, I expect the time to go up by 10x. The time for 100_000 is 12-15ms, the time for 1_000_000 is 88-94ms, so that's less than a factor of 10x. If any misinterpretation given your posted data, you should misinterpret that as 1_000_000 behaved better per-iteration than 100_000. (Really, the overhead is dwarfing everything else at that few iterations.) 1e6 to 1e7 shows <10x increase, so that's still okay; 1e7 to 1e8 shows almost exactly 10x increase, so that's right. 1e8 to 1e9 shows only 2x increase: this is the first where I'd think you were hitting a limit of some sort. What's the output of your `perl -V \| grep size`, especially `perl -V:ivsize`? Even with 4, I would have expected the 1e9 loop to increase linearly, because 1e9 only requires 30bits, which easily fits inside a 32-bit signed IV at 4-bytes. On my super-ancient system with a 4byte ivsize, `#/usr/bin/perl use warnings; use strict; use Benchmark qw(:hireswallclock timethese cmpthese); my $r = timethese( -60 , { oneE5 => sub { for( 1 .. 1e5 ) {} }, oneE6 => sub { for( 1 .. 1e6 ) {} }, oneE7 => sub { for( 1 .. 1e7 ) {} }, oneE8 => sub { for( 1 .. 1e8 ) {} }, oneE9 => sub { for( 1 .. 1e9 ) {} }, }); cmpthese($r);` [download] gives me oneE5: 63.876 wallclock secs (63.43 usr + 0.20 sys = 63.63 CPU) +@ 140.81/s (n=8960) oneE6: 63.7849 wallclock secs (63.09 usr + 0.23 sys = 63.32 CPU) + @ 14.17/s (n=897) oneE7: 63.6703 wallclock secs (63.05 usr + 0.21 sys = 63.26 CPU) + @ 1.42/s (n=90) oneE8: 62.9542 wallclock secs (62.30 usr + 0.20 sys = 62.50 CPU) + @ 0.14/s (n=9) oneE9: 74.5811 wallclock secs (73.81 usr + 0.26 sys = 74.07 CPU) + @ 0.01/s (n=1) (warning: too few iterations for a reliable count) Rate oneE9 oneE8 oneE7 oneE6 oneE5 oneE9 1.35e-02/s -- -91% -99% -100% -100% oneE8 0.144/s 967% -- -90% -99% -100% oneE7 1.42/s 10438% 888% -- -90% -99% oneE6 14.2/s 104829% 9738% 896% -- -90% oneE5 141/s 1042910% 97688% 9798% 894% -- [download] which looks highly linear to me, through the 1-billion (1e9). I don't know why your 1e9 is only 2x your 1e8, because I cannot repeat that. But my guess, as hinted above, is ivsize. Or maybe it's the c-ish loop that hippo pointed out; you could test that by also including a couple more rows in the cmpthese() with c-style loops for 1e8 and 1e9. Also, longer iterations (like 600sec instead of 60sec) will give more accurate results; this was just a proof of concept. update: after reading haukex's reply, and wondering where the 73.8s came from, I re-read your timing, and saw the "1m" prefix on the output.. I was only seeing the "13.828s", which confused me. Now that I see 1m13.828s, that's ~74s, which as haukex said, is reasonably linear. So my confusion is allayed, so my requests for ivsize info is unnecessary.	[reply] [d/l] [select]