Re: Foreach for array pruning? (Updated)

Updated conclusion:

You can save some ram (assuming you haven't got any to spare) at a penalty of 10% runtime.

my @array = 1 .. 1_000_000; 
## Note the above doubles mem. consumption.
## Half of which is immediately availble to the rest of the program.

# Ripple the good values through the array
my( $s, $d );
for( $s = $d = 0; $s < @array; $d++, $s++ ) { 
     $array[ $s ] %2 and $s++;;
      $array[ $d ] = $array[ $s ]; 
}; 
# and truncate it.
$#array = --$d;
[download]

Memory test code.

#! perl -slw
use strict;

our $SIZE ||= 1_000_000;

# Don't build the test array by assigning a list 
# cos that doubles the memory consumption which
# then serves to conseal the extra consumed by the grep

my @array;
push @array, $_ for 1 .. $SIZE;

printf 'Check ram before ripple'; <STDIN>;

#@array = grep not ($_ % 2), @array;

my( $s, $d );
for( $s = $d = 0; $s < @array; $d++, $s++ ) { 
     $array[ $s ] %2 and $s++, $array[ $d ] = $array[ $s ]; 
}; 
$#array = --$d;

printf 'Check ram after ripple'; <STDIN>;

die 'M_S: Mismatch:', scalar @array, " @array[ 0, -1 ]"
    unless @array == $SIZE / 2 and "@array[ 0, -1 ]" eq "2 $SIZE";

__END__
P:\test>371457-mem
Check ram before grep 21844 kb
Check ram after grep  35856 kb

P:\test>371457-mem
Check ram before ripple 21,844 kb
Check ram after ripple  21,860 kb
[download]

Benchmark

#! perl -slw
use strict;
use Benchmark qw[ cmpthese ];

our $SIZE ||= 1000;
our $DEBUG;

cmpthese( 5, {
    simple_grep => q[
        my @array = 1 .. $SIZE;

        @array = grep not ($_ % 2), @array;

        die 'S_G: Mismatch:', scalar @array, " @array[ 0, -1 ]"
            unless @array == $SIZE / 2 and "@array[ 0, -1 ]" eq "2 $SI
+ZE";
    ],
    splice_for => q[
        my @array = 1 .. $SIZE;

        $array[ $_ ] % 2 and splice @array, $_, 1 for reverse 0 .. $#a
+rray;

        die 'S_F: Mismatch:', scalar @array, " @array[ 0, -1 ]"
            unless @array == $SIZE / 2 and "@array[ 0, -1 ]" eq "2 $SI
+ZE";
    ],
    move_splice => q[
        my @array = 1 .. $SIZE;

        my( $s, $d );
        for( $s = $d = 0; $s < @array; $d++, $s++ ) { 
            $array[ $s ] %2 and $s++, $array[ $d ] = $array[ $s ]; 
        }; 
        $#array = --$d;

        die 'M_S: Mismatch:', scalar @array, " @array[ 0, -1 ]"
            unless @array == $SIZE / 2 and "@array[ 0, -1 ]" eq "2 $SI
+ZE";
    ],
});

__END__
P:\test>371457 -SIZE=100000
            (warning: too few iterations for a reliable count)

               Rate  splice_for move_splice simple_grep
splice_for  0.798/s          --        -94%        -94%
move_splice  12.3/s       1440%          --         -8%
simple_grep  13.3/s       1571%          9%          --

P:\test>371457 -SIZE=2000000  ## splice_for removed from test.
            s/iter move_splice simple_grep
move_splice   1.66          --         -8%
simple_grep   1.52          9%          --
[download]

Comment on Re: Foreach for array pruning? (Updated) Select or Download Code

Replies are listed 'Best First'.
Re^2: Foreach for array pruning? (Updated) by spandox (Novice) on Jul 02, 2004 at 21:43 UTC
I think that misses the point. It was more a general question with a "trivial" example BTW I was on perl 5.6.1 I installed 5.8.4 (and a boatload of modules - thanks CPAN) and did a time comparison `Code used: #!/usr/bin/perl use Time::HiRes qw(gettimeofday tv_interval ); use strict; # good habits start at birth my ($start,$stop); my @moo; @moo = (1 .. 1000000); $start = [gettimeofday()]; @moo = grep {; {if ($_ % 2 > 0){ 0;last; } 1;}} @moo; $stop = [gettimeofday()]; print tv_interval($start, $stop), "\n\n";` [download] results: 5.6.1 did pretty much the same as 5.8.4 PS: anyone figure out why I needed the ';' in between the squigle brackets for the code blocks?	[reply] [d/l]
Re^3: Foreach for array pruning? (Updated) by BrowserUk (Patriarch) on Jul 02, 2004 at 22:34 UTC
I'm not quite sure what you mean by "misses the point"? I thought you were looking for a way to avoid your percieved inefficiencies of using grep to filter large arrys. The only 'inefficiency' that really exists is that (notionally) a list is created on one side of the grep which is then filtered and assigned back to the array on the other. Ie. Perl trades space for time. Given the way Perl implements it's arrays, there is no trivial way of removing an element from the middle. `[@myarray]-->[ 0 ]->[ value 1 ] [ 1 ]->[ value 2 ] [ 2 ]->[ value 3 ] [ 3 ]->[ value 4 ] [ 4 ]->[ value 5 ]` [download] To remove a value from the middle (say element 2) in-place, you have to ripple the pointers. `[@myarray]-->[ 0 ]---->[ value 1 ] [ 1 ]---->[ value 2 ] [ 2 ]-\ [ 3 ]-\\->[ value 4 ] \->[ value 5 ]` [download] In effect (if not in exact implementation), is to copy pointers from the old list into a new one omiting those that fail the test. Then discard the old array if the destination is the same as the source. Very efficient cpu-wise but does require a extra memory. Doing it in-place avoids the extra memory, but there is no way to void the ripple. That's basically what my snippet did. It's obviously not going to be as fast, though I was surprised how little slower it was. Whether it would be possible to add an optimisation to grep to do this in line if the dest == src I'm not sure. I know that sort has a similar 'in-place' optimisation for dest == src invocations. With respect to the need for the ';'... It's required to allow the parser to determine that your inner code block `@moo = grep {; {if ($_ % 2 > 0){ 0;last; } 1;}} @moo;` [download] Is a code block and not an anonymous hash. More typically this is done with a unary '+' `@moo = grep { +{if ($_ % 2 > 0){ 0;last; } 1;} } @moo;` [download] Alternatively, you could avoid the need for the inner block completely and do `@moo = grep { not ( $_ % 2 ) } @moo;` [download] Examine what is said, not who speaks. "Efficiency is intelligent laziness." -David Dunham "Think for yourself!" - Abigail "Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon	[reply] [d/l] [select]
Re^4: Foreach for array pruning? (Updated) by spandox (Novice) on Jul 03, 2004 at 05:02 UTC
Why the inner code block? As mentioned before, and explained in the comments, this is a trivial example that is meant to mimic the behavior of a foreach with the "added feature" I discussed earlier The concept is to come up with something that could work with fairly complex code. Perhaps it is going to start with a list of servers - do some checks and remove from the array all of the servers that respond in a certain way when queried. This may take more than the trivial one line of code. "I" may also want to "fall out" of the code at many different points. That is why I made a version that had the "last" command in it. I did not mention in the code my "normal" procedure either `my @temp; foreach (@moo) {if (! ($_ % 2)){push @temp, $_;}} @moo=@temp;` [download] Which might I add, for an array or 1,000,000 elements benchmarks at around half of the time of any of the examples of that have an inner block (the last example above without the inner block cheats becuase it cannot be changed for any reasonable increase in code complexity ************** Why does the parser have trouble when it is in a grep? `my $i=1; {{{print "$i";}; if ($i++ < 5){redo;}}}` [download] Runs without hessitation, it is just nested code blocks for goodness sakes......and why can I do a last or redo within the codeblock of a grep - Why do I need to nest a second block in there? Just curious --Spandox	[reply] [d/l] [select]
Re^3: Foreach for array pruning? (Updated) by duff (Parson) on Jul 02, 2004 at 21:52 UTC
PS: anyone figure out why I needed the ';' in between the squigle brackets for the code blocks? At a guess I'd say it's so perl can disambiguate between a block of code and the start of a hash ref construction. duff	[reply]