I have been working on sorting a large text file by a datestamp within the file. My objective is to get it to run with a minimal memory footprint. The $offsets var has alternating byte offsets and epoch timestamps. I now need to sort them by timestamp. I know that my data structure is not well suited for this application, but due to the fact that I want to keep the memory utilization low, it's the best option. Once the timestamps are in order, I want to print out a sorted log based on the byte offset and timestamp relation. I am hopeing that someone can offer some perls of wisdom. Thanks 
#!/usr/bin/perl -w
use strict;
use Date::Calc qw(Mktime Today_and_Now Delta_DHMS);

my @starttime = Today_and_Now;

open (BIGLOG, "< E:/biglog.log") || die "Cannot open log\n";

my $offsets = pack 'I', 0; 
$offsets .= pack 'I', 0;

while (<BIGLOG>){
    $offsets .= pack 'V', tell BIGLOG;

    if ($_ =~ /^\s*#/ || $_ =~ /^\s\n/ || $_ !~ /^\s*\d/ ){
        $offsets .= pack 'V', 0;
    }
    else{
        $offsets .= pack 'V', Mktime(unpack("A4xA2xA2xA2xA2xA2", $_));
    }
    
}

my @endindex = Today_and_Now;
print "\nIndexed ". (length($offsets)/8 -1)." Lines in ";
printf("%02d Days, %02d Hours, %02d Minutes, %02d Seconds", Delta_DHMS
+(@starttime, @endindex));

# EPOCH = ($_ - 1) * 8 + 4;
# BYTE = ($_ - 1) * 8;

##
#
# Sort
#
##

my @endtime = Today_and_Now;

print "\nTotal runtime:\t\t";
printf("%02d Days, %02d Hours, %02d Minutes, %02d Seconds\n", Delta_DH
+MS(@starttime, @endtime));

close BIGLOG;
exit;

sub readline_n{ 
    my( $fh, $line) = @_; 
    seek $fh, unpack( 'V', substr( $offsets, ($line - 1) * 8, 4 )), 0;
    scalar <$fh> 
}

[download]

In reply to Sort big text file - byte offset - 50% there by msalerno

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.