Re: How to sort a large flat file 90mb with PERL-- various ways/tradeoffs/watchouts

I agree that native unix sort is probably the best idea. You can install modules see A Guide To Installing Modules for details on how to install locally for yourself. If you have memory to burn you could just slurp the file into an array and sort it using perl's sort:

#!/usr/bin/perl
open F, $ARGV[0] or die "Usage $0 infile > outfile\n";
@ary = <F>;
close F;
print for sort @ary;
[download]

This code will almost certainly be significantly slower and use a lot more memory than native unix sort. ~~Because we output the sorted results immediately we only have one array in memory.....~~

cheers

tachyon

Comment on Re: How to sort a large flat file 90mb with PERL-- various ways/tradeoffs/watchouts Download Code

Replies are listed 'Best First'.
Re^2: How to sort a large flat file 90mb with PERL-- various ways/tradeoffs/watchouts by theorbtwo (Prior) on Jun 30, 2004 at 07:16 UTC
Actually, you still have two arrays in memory. `print for (sort @ary);` creates a new array to hold the results of the sort, and can't throw out the value of @ary (until after the script ends), because the variable @ary is still in scope. `@ary=sort @ary` doesn't have this problem, in theory, but up to somewhen in the 5.8.x series (5.8.2, IIRC), it created a temporary array anyway. In any case, yes, native sort is probably faster then perl sort for this.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^2: How to sort a large flat file 90mb with PERL-- various ways/tradeoffs/watchouts
by theorbtwo (Prior) on Jun 30, 2004 at 07:16 UTC

Actually, you still have two arrays in memory. print for (sort @ary); creates a new array to hold the results of the sort, and can't throw out the value of @ary (until after the script ends), because the variable @ary is still in scope. @ary=sort @ary doesn't have this problem, in theory, but up to somewhen in the 5.8.x series (5.8.2, IIRC), it created a temporary array anyway.

In any case, yes, native sort is probably faster then perl sort for this.

[reply]
[d/l]
[select]