in reply to Sorting a (very) large file

There are three things you could try.:

1) The one that will always work is to split up the file in smaller chunks, sort the separately, and then merge them again.

2) If you work on a unixish system, you could try to abuse you system's sort program, at least GNU sort handles memory limitations automatically by creating temporary files.

3) Search on CPAN. Maybe there's a module that does what you want? File::Sort looks promising at a first glance (actually CPAN should be your first place to look ;-)

Replies are listed 'Best First'.
Re^2: Sorting a (very) large file
by salva (Canon) on Dec 01, 2007 at 09:21 UTC
    3) Search on CPAN. Maybe there's a module that does what you want?

    Sort::External

[reply]
      Y'know, Sort::External is designed for exactly what the original poster wants to do... but few people seem to find it on CPAN.

      Sort::External was my first CPAN distro, and the people on the modules mailing list prevailed upon me to release it it under the academically correct name. I occasionally regret having taken that advice.

      FWIW, here's the definition of "external sort" from nist.gov:

      Any sort algorithm that uses external memory, such as tape or disk, during the sort. Since most common sort algorithms assume high-speed random access to all intermediate memory, they are unsuitable if the values to be sorted don't fit in main memory.
      --
      Marvin Humphrey
      Rectangular Research ― http://www.rectangular.com
[reply]
        but few people seem to find it on CPAN

        On search.cpan.org, searching for "sort" returns almost 2000 hits, so it's hardly surprising that people doesn't find your module :-(

[reply]

In Section Seekers of Perl Wisdom