in reply to sort large file

Use the sort command, it's optimized to sort large files. It can figure out when to sort in memory, and when to use temporary files. It's a standard Unix tool, and has been ported to different OSses, including Windows.

Abigail

Replies are listed 'Best First'.
Re: Re: sort large file
by z3d (Scribe) on Apr 01, 2004 at 15:33 UTC
    ...unless the file is truly huge. Sort has its limitations - for instance, on solaris it tends to use /var/tmp to store that temporary file, and if you're dealing with a few gigs worth of data, you can easily run out of even disk space for the tmp files (painful, true story). Not that I have a better solution :)



    "I have never written bad code. There are merely unanticipated features."
      Solaris sort offers -T to specify a directory for its temprary files.
      Sure, there will always be limitations. But it's usually cheaper/easier to add a few gigs of disk space than it is to add a few gigs of RAM. (And if you do it in Perl, multiply that by a factor as well). Typically, 'sort' will need free disk space about the same amount as the file to sort is large.

      Abigail