in reply to character-by-character in a huge file

Consider the informal benchmark
use Benchmark qw|:all|; my @a = (1..1_000_000); my $b; my $c = 'x' x 1_000_000; my $d; cmpthese( 5, { ary => sub { for my $i (0..1_000_000-501) { $b = $a[$i] - $a[$i+500]; } }, substr => sub { for my $i (0..1_000_000-501) { $d = substr($c,$i,1) . substr($c,$i+500,1); } }, unpack => sub { my @b = unpack 'c*', $c }, split => sub { my @b = split //, $c } } );
On my arthritic PIII machine, I get
Benchmark: timing 5 iterations of ary, split, substr, unpack... ary: 8 wallclock secs ( 8.62 usr + 0.00 sys = 8.62 CPU) @ 0 +.58/s (n=5) split: 26 wallclock secs (24.93 usr + 0.33 sys = 25.26 CPU) @ 0 +.20/s (n=5) substr: 11 wallclock secs (11.64 usr + 0.00 sys = 11.64 CPU) @ 0 +.43/s (n=5) unpack: 7 wallclock secs ( 6.14 usr + 0.03 sys = 6.17 CPU) @ 0 +.81/s (n=5) s/iter split substr ary unpack split 5.05 -- -54% -66% -76% substr 2.33 117% -- -26% -47% ary 1.72 193% 35% -- -28% unpack 1.23 309% 89% 40% --
This says that, for the ops between elements $i and $i+500 I picked, an array rep is faster than a string rep, but the overhead of breaking a string into an array will negate that advantage. The best you could do on my machine with these algorithms and ops it is to move along the string at about 500_000 chars/sec.

Another solution that has not been mentioned yet is using a regexp. From perlretut:

while ($dna =~ /(?=A.{500}T)./g) { print "Got an AT match at position ", pos $dna, "\n"; }
Update Fixed regexp.

-Mark