When saving a fixed amount of incoming data and throwing it away most of the time as new data comes in, (eg, saving n preceding lines while(<>) processing files), this little gadget can be twice as fast as shift/push.
update:The code to read the array is simplified thanks to fundflow.
I originally posted this just as a novelty. But it seems to have turned out to be an interesting demonstration of the speed-up that can come from instruction locality (see below).
$sz ||= 3; # user parameter $top = $sz - 1; @next = ( 1 .. $top, 0 ); $tl = $top; @a = (); # . . . $a[ $tl = $next[$tl] ] = $val; # 'push' zero or more times # . . . @all = @a[ $tl + 1 .. $#a, 0 .. $tl ]; $tl = $top; @a = ();

Replies are listed 'Best First'.
Re: Circular buffer instead of shift/push
by repson (Chaplain) on Jan 12, 2001 at 09:53 UTC
    Here's much the same thing, but using a closure. I'm sure this will slow down the process but can improve the usage of it by seperating the buffer from the rest of the code, and allowing easy reuse - and multiple buffers if desired - without needing to intersperse your code in amoungst other processing. It's a matter of tradeoffs, but I'm not sure on how much slower it will be.
    my $buf = make_buf(3); &$buf('hi'); &$buf('moo'); print join(' ', &$buf) . "\n"; # 'hi moo' $buf->('foo','no'); print join(' ', $buf->()) . "\n"; # 'moo foo no' sub make_buf { my $size = shift || 3; my @buf; my $pos = -1; my $last = $size - 1; return sub { if (@_) { while (@_) { $buf[ $pos==$last ? $pos=0 : ++$pos ] = shift + } } elsif (wantarray) { return ($#buf==$last ? @buf[ $pos+1 .. $last, 0 .. $pos ] +: @buf); } else { warn "Call with an argument or in list context"; return +; } } }
      Nice. Note that you can change the actual operations inside the sub to those suggested above. Removing the trinary operator ?: will definitly help speed.
Re: Circular buffer instead of shift/push
by fundflow (Chaplain) on Jan 12, 2001 at 05:24 UTC
    Nice.

    I would change the main part to: (using your terms)

    $a[ $tl++ % $sz ] = $val
    or
    $a[ $tl=($tl+1)% $sz ] = $val
    This replaces a memory access with cheap operator.

    The last line can be written as:

    @all=@a[ $tl..($sz-1), 0..($tl-1) ];
    (assuming the array is full, othewise add your check)
      Thank you! Your right, of course, about unloading the array.
      (I guess I was just using the index array again because I had it.)
      Note that it would actually be:
      @all = @a < $sz ? @a : @a[ $tl+1 .. $top, 0 .. $tl ];
      since $tl ("tail") is left pointing at the last item written.

      About replacing the memory access with a modulo operation:
      Maybe. But modulo ain't cheap and at least for small $sz's,
      the memory access may not be one (caches). The whole point of
      the exercise was to avoid the slowdown from the
      shift @a if @a == $sz;
      test in the canonical loop. Modulo would seem to be in the same
      class. On the Solaris I'm working on at the moment, the indexed
      index is much faster than modulo, at least up to arrays of 3000.
      In fact, the modulo is only marginally faster than shift/push.

      Oddly, the direct retrieval seems to take about as long as retrieving
      through the index array(!?!), but it certainly is clearer,
      thanks again,


      p
        Note that modulus can be very cheap sometimes (namely in 2 powers, using AND)

        If you have time, it will be interesting to know if it actually makes a difference.

Re: Circular buffer instead of shift/push
by petral (Curate) on Mar 10, 2001 at 12:16 UTC
    This was my first attempt to post on PerlMonks. The main lesson I got from this fiasco is that you don't just need to find the time to post, but also post when you'll have time to respond for a day or two.

    Anyway, the following benchmarks are typical of quite a few I ran (including large and huge sizes of arrays). The dramatic thing is that even though indexing into an array (called "wheel" here in the first set) is not especially fast, indexing into an array to increment the index while indexing into an array, is faster than any other way of stepping one step. The only explanation I can see, is that the code for indexing is already in the instruction cache. Any other method, by definition, requires two unrelated series of instructions to be loaded.
    (Why the "pow2modix" slows down _so_ much, I don't know. update: because the first(fast) one is optimized away in void context?)

    Anyway, here's some benchmarks and, no, I won't be able to respond quickly to comments, but will eventually. (It's so "stale" already, that further slow response, if there are comments, probably won't be noticed.)
    $ uname -a SunOS xxxxxx 5.6 Generic_105181-15 sun4u sparc SUNW,Ultra-2 $ perl -v This is perl, version 5.005_03 built for sun4-solaris $ perl -MBenchmark -e 'my $m=3; my ($xe,$xp,$xa,$xw)=($m)x4; my @n=(1..$m,0); my @a=("")x$m+1; timethese(-5, { ashiftp => q{ shift @a if $#a==$m }, equaland => q{ $xe < $m and ++$xe or $xe = 0 }, pow2and => q{ scalar(++$xp, $xp &= $m) }, pow2mod => q{ ++$xp & $m }, wheel => q{ $xw = $n[$xw] }, } );' | perl -nawe '$.==1 and print and next;($x=$F[14])=~s/\/.*//;$b||=$x; printf"%12s %12s %5.2fusecs %.2f\n",@F[0,14],1000000/$x,$b/$x' - perl -MBenchmark -e 'my $m=3;my($x,$xe,$xa,$xp,$xw)=($m)x5;my @n=(1..$ +m,0); my (@a,@ai,@ae,@aa,@ap,@aw); @a=@ai=@ae=@aa=@ap=@aw=("")x +$m+1; timethese(-5, { ashpush => q{ shift @a; push @a, $x }, ashifpush => q{ shift @ai if $#ai == $m; push @ai, $x +}, equalandix => q{ $ae[ $xe < $m and ++$xe or $xe = 0 ] = + $xe }, pow2andix => q{ $aa[ ++$xa, $xa &= $m ] = $xa }, pow2modix => q{ $ap[ ++$xp & $m ] = $xw }, wheelix => q{ $aw[ $xw = $n[$xw] ] = $xw }, } );' | perl -nawe '$.==1 and print and next;($x=$F[14])=~s/\/.*//;$b||=$x; printf"%12s %12s %5.2fusecs %.2f\n",@F[0,14],1000000/$x,$b/$x' - > > > > > > > > Benchmark: running ashiftp, equaland, pow2and, pow2mod +,\ wheel, each for at least 5 CPU seconds...\ ashiftp: 280765.34/s 3.56usecs 1.00 equaland: 341119.76/s 2.93usecs 0.82 pow2and: 368781.20/s 2.71usecs 0.76 pow2mod: 568666.50/s 1.76usecs 0.49 wheel: 451163.20/s 2.22usecs 0.62 $ > > > > > > > > > > Benchmark: running ashifpush, ashpush, equalandi +x,\ pow2andix, pow2modix, wheelix, each for at least 5 CPU seconds...\ ashifpush: 115283.93/s 8.67usecs 1.00 ashpush: 223748.51/s 4.47usecs 0.52 equalandix: 198690.04/s 5.03usecs 0.58 pow2andix: 209246.30/s 4.78usecs 0.55 pow2modix: 212778.91/s 4.70usecs 0.54 wheelix: 255846.92/s 3.91usecs 0.45 $ uname -a Linux xxxxxx 2.2.14 #1 SMP Mon Mar 27 16:46:10 EST 2000 i686 unknown $ perl -v This is perl, version 5.005_03 built for i386-linux Benchmark: running ashiftp, equaland, pow2and, pow2mod, wheel,\ each for at least 5 CPU seconds...\ ashiftp: 886579.73/s 1.13usecs 1.00 equaland: 1030440.00/s 0.97usecs 0.86 pow2and: 1309018.03/s 0.76usecs 0.68 pow2mod: 1750427.08/s 0.57usecs 0.51 wheel: 1243260.48/s 0.80usecs 0.71 Benchmark: running ashifpush, ashpush, equalandix, pow2andix, pow2modi +x,\ wheelix, each for at least 5 CPU seconds...\ ashifpush: 237115.60/s 4.22usecs 1.00 ashpush: 437152.00/s 2.29usecs 0.54 equalandix: 548898.80/s 1.82usecs 0.43 pow2andix: 514532.10/s 1.94usecs 0.46 pow2modix: 485667.44/s 2.06usecs 0.49 wheelix: 645636.33/s 1.55usecs 0.37


    p