Adding a lot of small integers

kappa has asked for the wisdom of the Perl Monks concerning the following question:

Update down there

Good day, fellow monks.

My question is more of academic interest than practical. Here's a simple task from an old CS collegiate contest: Sums in a Triangle.

It's definitely NOT of the type Perl excels at but still I decided to give it a try. (Actually, my main motivation is that a working Python solution exists!)

My C solution works fast and passed the online judge, it's O(n^2) and very efficient. But the best Perl code I came up with requires at least 6 seconds in the worst case on my fairly powerful Core 2 Duo 2.66Ghz box. It uses the same algorithm I used in C.

If anyone feels like playing with it, I'd be very interested in results.

My C code (before compressing into 256 chars which is not interesting):

#include <stdio.h>

int data[100][100];

int sumintr(int nr) {
    int r, col;
    for(r = nr - 1 - 1; r >= 0; --r) {
        for(col = 0; col <= r; ++col) {
            data[r][col] += data[r + 1][
              data[r + 1][col] > data[r + 1][col + 1]
                ? col
                : col + 1
              ];
        }
    }

    return data[0][0];
}

int main() {
    int ns, nr, s, r, col;

    scanf("%d\n", &ns);
    for(s = 0; s < ns; ++s) {
        scanf("%d\n", &nr);
        for(r = 0; r < nr; ++r) {
            for(col = 0; col <= r; ++col)
                scanf("%d", &data[r][col]);
        }

        printf("%d\n", sumintr(nr));
    }
}
[download]

My best Perl code (strict was removed):

#!/usr/bin/perl

for (1..<>) {
    @tr =
        map { [split ' ', <>] } 1 .. <>;

    $prev = pop @tr;
    for $cur (reverse @tr) {
        $g = $prev->[0];
        for (0 .. $#$cur) {
            $cur->[$_] += ($f = $g) > ($g = $prev->[$_ + 1])
                ? $f
                : $g;
        }

        $prev = $cur;
    }

    print "$prev->[0]\n";
}
[download]

The worst test case I'm banging my head against: http://dl.dropbox.com/u/360558/input_big.txt.gz (will decompress to 14Mb)

(Probably, my first reaction to such a question would be something like: "Wow, compiled C code is faster than Perl! Thank you, Captain Obvious!" I would definitely not spend several hours examining NYTProf beautiful reports unless I knew that some guy solved it in Python (albeit using a lot of memory, which is, by the way, a good indicator of a different algorithm)).

Update. Big thanks to ikegami and zwon for their suggestions in comments! I just decreased my runtime by almost 30% implementing them! The last version looks like this:

#!/usr/bin/perl

undef $/; @file = split ' ', scalar <>;

for (1 .. shift(@file)) {
    $rows = $file[$cur++];

    @s = (0) x ($rows + 1);
    for (1 .. $rows) {
        ($i, $y) = (0, 0);
        $cur += $_ - 1;

        for ($cur .. $cur + $_ - 1) {
            $x = $y;
            $y = $s[$i];
            $s[$i++] = $file[$_] + ($x > $y ? $x : $y);
        }
    }

    $m = 0;
    $m = $m > $_ ? $m : $_ for @s;
    print "$m\n";

    $cur += $rows;
}
[download]

It does not pass the SPOJ judge yet, although. But the box which will run it in 2 sec should be only twice as fast as my desktop :)

--kap

Comment on Adding a lot of small integers Select or Download Code

Replies are listed 'Best First'.
Re: Adding a lot of small integers by ikegami (Patriarch) on Jan 09, 2010 at 18:17 UTC
First, I see no reason to avoid strict and warnings. The only overhead it adds is the clearing of the variables in each loop pass. If that's an issue, you can avoid the clearing by declaring the vars outside the loops. IO carries a lot of overhead. Do it all at once. (The python player probably did the same thing, thus the high memory usage.) `#!/usr/bin/perl use strict; use warnings; my (@file, @tr, $prev, $cur, $f, $g); { local $/; @file = split ' ', scalar(<>); } for (1..shift(@file)) { ($prev, @tr) = reverse map [ splice(@file, 0, $_) ], 1..shift(@file); for $cur (@tr) { $g = $prev->[0]; for (0 .. $#$cur) { $cur->[$_] += ($f = $g) > ($g = $prev->[$_ + 1]) ? $f : $g; } $prev = $cur; } print "$prev->[0]\n"; }` [download] And then, let's avoid building an AoA. It saves on building time, memory costs, and avoids the need to deref. Note that splicing/poping/shifting from the ends of an array is very efficient. `#!/usr/bin/perl use strict; use warnings; my (@file, $rows, @tr, @prev, @cur, $f, $g); { local $/; @file = split ' ', scalar(<>); } for (1..shift(@file)) { $rows = shift(@file); @tr = splice(@file, 0, $rows($rows+1)/2); @prev = splice(@tr, -($rows--)); while ($rows) { @cur = splice(@tr, -($rows--)); $g = $prev[0]; for (0 .. $#cur) { $cur[$_] += ($f = $g) > ($g = $prev[$_ + 1]) ? $f : $g; } @prev = @cur; } print "$prev[0]\n"; }` [download] It might be faster to calculate the index into @tr instead of copying the values into in and out. But maybe not. The hindered readability and the extra additions might cancel out the benefit. `#!/usr/bin/perl use strict; use warnings; my (@file, $rows, @tr, $prev, $cur, $f, $g); { local $/; @file = split ' ', scalar(<>); } for (1..shift(@file)) { $rows = shift(@file); @tr = splice(@file, 0, $rows($rows+1)/2); $prev = @tr - $rows--; while ($rows) { $cur = $prev - $rows--; $g = $tr[$prev]; for (0 .. $rows) { $tr[$cur + $_] += ($f = $g) > ($g = $tr[$prev + $_ + 1]) ? $f : $g; } $prev = $cur; } print "$tr[0]\n"; }` [download] You could do something similar to avoid creating @tr. By the way, `($f = $g) > ($g = $prev->[$_ + 1])` is not guaranteed to be evaluated in the order in which you want it to be evaluated. It's technically a bug since operand evaluation order is undefined (although predictable and unlikely to change) in Perl.	[reply] [d/l] [select]
Re^2: Adding a lot of small integers by kappa (Chaplain) on Jan 09, 2010 at 23:28 UTC
Thank you for comprehensive comment, I'm jumping right in to try your suggestions! First, I see no reason to avoid strict and warnings. The only overhead it adds is the clearing of the variables in each loop pass. If that's an issue, you can avoid the clearing by declaring the vars outside the loops. Adding `use strict;` and declaring all vars right at the top with `our` increases my runtime by 5%. I didn't know about strict clearing the variables, are you sure this is the reason? --kap	[reply] [d/l] [select]
Re^3: Adding a lot of small integers by ikegami (Patriarch) on Jan 10, 2010 at 08:17 UTC
Adding use strict; and declaring all vars right at the top with our increases my runtime by 5%. That's surprising, but why would use `our`?	[reply] [d/l]
Re^2: Adding a lot of small integers by kappa (Chaplain) on Jan 10, 2010 at 00:11 UTC
I completely eliminated all operations on the structure of data by using indices to a flat `@file` array all over the place as per your final suggestion and it's the fastest version so far :) Thanks a lot! --kap	[reply] [d/l]
Re^2: Adding a lot of small integers by JavaFan (Canon) on Jan 09, 2010 at 23:07 UTC
First, I see no reason to avoid strict and warnings. The only overhead it adds is the clearing of the variables in each loop pass. No, it doesn't. Neither strict nor warnings makes that any variables are cleared. Variables may be cleared if you define them as lexicals inside a loop - but that doesn't change whether you have 'use strict; use warning;' in your program or not.	[reply]
Re^3: Adding a lot of small integers by ikegami (Patriarch) on Jan 10, 2010 at 08:16 UTC
I know. Did you read the rest?	[reply]
Re^4: Adding a lot of small integers by JavaFan (Canon) on Jan 10, 2010 at 11:05 UTC
Re: Adding a lot of small integers by zwon (Abbot) on Jan 09, 2010 at 20:25 UTC
On my C2D 2GHz your code is executed in 4.4s with 5.10.1 and in 4.0s with 5.8.9. The following code takes about 3.7s: Read more... (578 Bytes) And this seems slightly faster: `for ( 1 .. <> ) { @s = (); for ( 1 .. <> ) { @l = split ' ', <>; ( $i, $y ) = ( 0, 0 ); for (@l) { $x = $y; $y = $s[$i]; $s[ $i++ ] = $x > $y ? $_ + $x : $_ + $y; } } $m = 0; $m = $m > $_ ? $m : $_ for @s; print "$m\n"; }` [download]	[reply] [d/l] [select]
Re^2: Adding a lot of small integers by kappa (Chaplain) on Jan 10, 2010 at 01:07 UTC
Thanks, I like this top-down approach! Somehow, it didn't come to me in the first place. Btw, I managed to squeeze some more time from your second variant by replacing `@s = ()` with `@s = (0) x 100`. Benefit from pre-allocating memory for all elements, I suppose. --kap	[reply] [d/l] [select]
Re: Adding a lot of small integers by spx2 (Deacon) on Jan 09, 2010 at 18:00 UTC
you're using `pop` in your Perl version. In your C solution you're just using indexes and the array, why don't you do exactly the same in Perl and then report back with some time measurements.	[reply] [d/l]


Welcome to the Monastery
	PerlMonks