The facts I've found so far:

A possible strategy would be to first 'index' the file, by reading it line by line;

(untested code follows)
my %index; my $line=0; while (<FILE>) { my ($location,$time) = /^..(.{9})(.{4})/; push @{$index{$location}},[$time,$line]; $line++; }

This would results in a hash keyed on the 'location', with the value being a reference to an array with contain the info you need to sort the lines. This seems to be the minumum amount of info needed to determine the sort order.

The next step is to sort the arrays by the time values, you've stored, and fetch the lines in order from the file:

(untested code again)
for my $location (keys %index) { my @sorted = sort { $a->[0] <=> $b->[0]} @{$index{$location}}; for my $entry (@sorted) { seek FILE, 81 * $entry->[1], 0; read FILE, $line, 80; print $line,"\n"; } }

This method should be very memory efficient I think, and not too slow either; the biggest slowdown is probably the seeking around in the file.

This method works because we know the lengths of records. If we don't we could use the tell function before we read a line, to also store the exact start position of the line in the index...


In reply to Re^3: Reducing Memory Usage by Gilimanjaro
in thread Reducing Memory Usage by PerlingTheUK

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.