in reply to Re: EBCDIC sort
in thread EBCDIC sort

Esteemed Monks,

Thanks for the suggestionsSorry for not formatting the question properly and not listing my code.

The code works but I am facing another problem.

I am sorting a file that it more than 400MB size and using multiple keys. While executing, the sort fails with core dump and sometimes illegal instruction message.

I did some investigation and found out that it is due to insufficeint memory.

I am running it on AIX server and when I did ulimit -a it gave memory as 65536 bytes. I have unlimited file size permission. I reduced the size of the input file to about 60 K and the sort worked. I have checked and I can't use malloc() or reset it to be used during runtime.

use strict; use Encode qw(encode decode); ### Define the sort key here ### # Sorts in ascending order. sub key1 { ( substr( $a, 3, 17 )) cmp ( substr( $b, 3, 17 )); } # Sorts descending order. sub key2 { ( substr( $b, 20, 2 )) cmp ( substr( $a, 20, 2 )); } # ### Sort processing starts ### my @infile = <>; # Reads file ### Multiple sort keys can be defined and sorted in the order of t +he key my @sorted = map { decode('cp1047', $_) } sort { key1 || key2 } (m +ap { encode('cp1047', $_) } @infile); print @sorted;
Is there a more efficient way to reduce memory usage?

Replies are listed 'Best First'.
Re^2: EBCDIC sort (/bin/sort)
by tye (Sage) on Jun 05, 2007 at 18:13 UTC

    Do a GRT through /bin/sort.

    # ... use IPC::Open2 'open2'; my( $chaos, $sorted ); my $pid= open2( $sorted, $chaos, 'sort' ); while( <> ) { my $ascii= encode( 'cp1047', $_ ); print $chaos substr($ascii,3,17), substr($ascii,20,2), $_; } close $chaos; while( <$sorted> ) { print substr( $_, 19 ); }

    or thereabouts. It looks like your EBCDIC file is using ASCII newlines, which simplifies things.

    - tye