Sorting files

kepler has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I have several files, which I want to sort in a specific way, but all sumed up together. First, I've opened one file, one at the time, and added the lines to a single array. Then made a sort - it all worked fine. But now, the data is to large for the array I think - I get an "out of memory" error. Any suggestions?

Regards,

Kepler

Comment on Sorting files

Replies are listed 'Best First'.
Re: Sorting files by Corion (Patriarch) on Oct 02, 2015 at 07:47 UTC
Usually what you can do is to sort as many files as you can in memory and write those temporary files to disk. Merging sorted lists is something that can be done in constant memory by reading a line from each file and then writing the smallest line of these into a new file, refilling from the file. There also are Sort::Merge and Sort::External I believe.	[reply]
Re: Sorting files by Anonymous Monk on Oct 02, 2015 at 07:53 UTC
sure, decorate/sort/undecorate, see Understanding the Schwartzian transform., sorting type question- space problems basically, tag every line with a calculated key ... for example turn 1.3 and x1.1y into 000001.000003 and x000001.000001y ... whatever makes the sense for the order you want Then use the gnu sort or some other sort to sort your giant file	[reply]
Re: Sorting files by SuicideJunkie (Vicar) on Oct 02, 2015 at 13:48 UTC
You should also make sure you're not making unnecessary copies of the array(s), such as when calling functions. Duplication will drastically reduce the amount of data you can hold before you run out of memory;)	[reply]
Re: Sorting files by locked_user sundialsvc4 (Abbot) on Oct 02, 2015 at 15:51 UTC
A short snippet of the code that you are using ... in code-tags, as described in Markup in the Monastery ... would help us give you better answers. Also, more details about the files: how many files, and how large each?