Are there other techniques I should use to traverse a large file like this and which might offer methods to move forward, back, go to beginning, etc.?

See seek. It works best if the records are fixed length. If they are not then creating an index that maps record number to file position is very simple and makes for quite fast access.

I have a file that is a 3.6GB and contains 40e6 records. I index it like this:

perl -e"BEGIN{binmode STDOUT}" 
-ne"print pack'Q',tell STDIN" syssort >syssort.idx
[download]

Which takes just a couple of minutes to run. I can then randomly access the records in that file using:

#! perl -slw
use strict;
use Time::HiRes qw[ time ];

our $N //= 1000;

open IDX, '+<:raw', 'syssort.idx' or die $!;
open DAT, '+<:raw', 'syssort' or die $!;

my $start = time;
for ( 1 .. $N ) {
    my $recnum = int rand 40e6;
    seek IDX, $recnum *8, 0;
    my $idx;
    read IDX, $idx, 8;
    my $pos = unpack 'Q', $idx;
    seek DAT, $pos, 0;
    chomp( my $record = <DAT> );
#    printf "Record %d: '%s'\n", $recnum, $record;
}
my $elapsed = time - $start;
printf "$N random records read in %.3f seconds (%6f/s)\n",
    $elapsed, $elapsed / $N;

__END__
c:\test>syssort-idx -N=1e4
1e4 random records read in 2.223 seconds (0.000222/s)

c:\test>syssort-idx -N=1e5
1e5 random records read in 21.332 seconds (0.000213/s)

c:\test>syssort-idx -N=1e3
1e3 random records read in 0.218 seconds (0.000218/s)

c:\test>syssort-idx -N=1e3
1e3 random records read in 0.226 seconds (0.000226/s)
[download]

At 0.2 milliseconds per record, it is fast enough for most purposes.

The seek function allows you to move forward and backward (and there's a complementary tell function which lets you know where you are in the file). However, unless you know exactly where you want to go, that's not necessarily going to be of much use. If you have fixed-length records and know exactly which record you're after, that may be a solution.

As already stated, a DB solution may be better.

BIGtiny:

I like the Count's suggestion to use a database, but if you want to use variable length records, you may also consider a modification of kcott's suggestion: If you have a key field, scan your file once, gathering the keys and file positions of the records, and construct a hash. Then you can query the hash table to seek your records quickly.

...roboticus

Not advocating that this is any better than previous suggestions, but you can check out Tie::File, which ties a file to an array where each element corresponds to a line in the file.

I’ll echo CountZero’s comment about SQLite.   That tool truly is a “game changer” when it comes to doing the things for which we require “flat files.”   If you have not yet looked at it very closely, you owe it to yourself (and to your projects) to do so A.S.A.P...