Like are anonymous monk above, I'm have great difficulty in seeing how this code is consuming large quantities of memory.

The only possibility I see is if the array of filenames you are passing by reference in $infile_aref is huge?

If this is the case, then passing by ref is a goood thing, but you immediately undo that by duplicating it's contents into a local (my'd) array along with several others:

    my @keyMap=@{$keyMap_aref};
    my @infile=@{$infile_aref};
    my $outfile=$outdir.'/'.$outFileName;
    my @keyPos=@{$keyPos_aref};
    my @outCol=@{$outCol_aref} if defined $outCol_aref;
[download]

The only benefit of which I can see is that it allows you to write

foreach my $infile(@infile){
[download]

Instead of

foreach my $infile( @{$infile_aref} ){
[download]

If these arrays are not very big, then that won't be the source of your problem, but it's the only possibility I can see. Otherwise, the memory consumption is occuring outside the auspices of the code you've shown us.

As far as performance is concerned. All your cpu's are benefiting you very little as the data transfer is going to a single file, through a very small cache (unless you have already increased this?). As such, your process is likely IO bound, only utilising a single cpu, and spending most of it's time in IO_wait states.

Frankly, building a B-tree of all your data just in order to sort it, given that you are then writing the sorted data back to a flat file, makes no sense at all. B-trees are very fast for searching an traversing, but they are expensive to build. Especially if the ordering function requires a call-back into perl code and utilises a multi-level comparison to boot.

You would be sustantially better off using a merge sort algorithm if your individual files are small enough to allow them to be sorted entirely in memory.

Though, if your system sort can handle the record lengths and volumes of data you are dealing with, and the embedded newlines is the only problem, then converting those embedded newlines to some 'otherwise non-occuring token', using the system sort, and converting the tokens back after makes the most sense of all.