in unix I cannot configure the input record separator. So if the data comes with embedded \n charcters in the middle then this breaks the sort.

Is it essential that you preserve these "embedded \n" characters? Whatever the answer, if you can get the unix sort process to create exactly the ordering you want, you could use Perl just to "normalize" the records so that they are all single-line and well-behaved when they go through the unix sort. (But unless you're using Gnu sort, you might still have a problem if some of the records end up being too long -- I think some flavors of unix sort may still have a limit of 1024 bytes per line). (update: For some reason, I feared that solaris might be one such limited flavor, but I was wrong -- I could pump lines of >8200 bytes through /usr/bin/sort with no loss of data. Still, if you're not on linux or solaris 8 or better, test it first.)

You are already handling record-based input by setting $/ in perl, so why not try a pipeline like this (I'm not sure if your reference to "164" was a decimal or octal value -- best to use hex and not worry about this ambiguity; I'll guess that you meant decimal):

perl -pe 'BEGIN {$/="\xa4\n"} s/(?<!\xa4)\n/ /g' | sort ...
[download]

Like are anonymous monk above, I'm have great difficulty in seeing how this code is consuming large quantities of memory.

The only possibility I see is if the array of filenames you are passing by reference in $infile_aref is huge?

If this is the case, then passing by ref is a goood thing, but you immediately undo that by duplicating it's contents into a local (my'd) array along with several others:

    my @keyMap=@{$keyMap_aref};
    my @infile=@{$infile_aref};
    my $outfile=$outdir.'/'.$outFileName;
    my @keyPos=@{$keyPos_aref};
    my @outCol=@{$outCol_aref} if defined $outCol_aref;
[download]

The only benefit of which I can see is that it allows you to write

foreach my $infile(@infile){
[download]

Instead of

foreach my $infile( @{$infile_aref} ){
[download]

If these arrays are not very big, then that won't be the source of your problem, but it's the only possibility I can see. Otherwise, the memory consumption is occuring outside the auspices of the code you've shown us.

As far as performance is concerned. All your cpu's are benefiting you very little as the data transfer is going to a single file, through a very small cache (unless you have already increased this?). As such, your process is likely IO bound, only utilising a single cpu, and spending most of it's time in IO_wait states.

Frankly, building a B-tree of all your data just in order to sort it, given that you are then writing the sorted data back to a flat file, makes no sense at all. B-trees are very fast for searching an traversing, but they are expensive to build. Especially if the ordering function requires a call-back into perl code and utilises a multi-level comparison to boot.

You would be sustantially better off using a merge sort algorithm if your individual files are small enough to allow them to be sorted entirely in memory.

Though, if your system sort can handle the record lengths and volumes of data you are dealing with, and the embedded newlines is the only problem, then converting those embedded newlines to some 'otherwise non-occuring token', using the system sort, and converting the tokens back after makes the most sense of all.

This reply does not solve OP's problem, but... A bit of white space would have been much appreciated in the first line of btree() (which assigns TONS of variables) and the $pcode assignment. Something like...

sub btree
{ #  I personally would much prefer hash reference(s) here
  my ( $disk , $workdir , $outFileName
      , $sep
      , $keyMap_aref
      , $verbose
      , $infile_aref , $indir , $outdir
      , $splitSize
      , $keyPos_aref , $outCol_aref
      ) = @_ ;

  ...

  #  No need for "$pcode" & later "eval $pcode"
  #  Use "$parse->($line)" as you already have
  my $parse =
    make_parse_code( $sep , $splitSize , $keyPos_aref );

  ...
}

sub make_parse_code
{ my ($sep , $size , $items) = @_;
  return
    sub
    { my ($line) = @_;
      join $sep
        , ( (split /$sep/ , $line , $size)[@$items] );
    };
}
[download]

...adjust to your liking. Well, lack of white space was bugging me much, so i adjusted the code; i am sure others [cw]ould point out other improvements/suggestions.

Try upgrading the db version and recompiling DB_File as well as increasing the cache to alleviate the speed issue (you can't do too much, cause file-io is only so fast).

If you don't need portability to platforms that don't support using 'sort', I'd suggest following graff's advice about transforming the data and then using Unix's sort command. This kind of implementation is straightforward, quick, and generally robust.