in reply to Re: Sort large files
in thread Sort large files

What I am not sure about is the maximum size. Sort::Maker appeared to be the closest match to the OP needs of the namespace you refer to, but even that didn't make it clear enough to me what the filesize limits are for the OP's particular key definitions for a particular CPAN module. The OP is partly at fault there for not giving that much info in regard to the distribution of information - two example records ain't much to go on.
__________________________________________________________________________________

^M Free your mind!

Replies are listed 'Best First'.
Re^3: Sort large files
by salva (Canon) on Jun 06, 2007 at 15:19 UTC
    Sort::Maker memory requirements are quite high so I don't think it could be a good solution.

    Sort::External does the sorting on disk, so it is not limited by the memory size and it is very easy to use for simple cases but for complex cases a transformation similar to the GRT is required.

    Another way is to use the external sort program that uses on-disk sorting algorithms:

    # untested ... use Encode qw(encode decode); use MIME::Base64 qw(encode_base64); my $tempfn = "sort.temp"; # better use File::Temp! open my $tmp, ">", $tempfn or die ...; while(<>) { my $k0 = encode_base64(encode(cp1047 => substr($_, 3, 17)), ""); my $k1 = encode_base64("\xff\xff" ^ encode(cp1047 => substr($_, 20, +2))); print $tmp join("\0", $k0, $k1, $_); } close $tmp or die "..."; open my $sorted, "-|", sort => $tempfn or die "..."; while(<$sorted>) { print((split /\x00/, $_, 3)[2]); }
      I was going to say, yes, but what about sort keys, until I just-in-time found it explained in Sort::External::Cookbook - so I now agree with you!
      __________________________________________________________________________________

      ^M Free your mind!