Re: Saving an array to a disk file

Rather than writing the array to disk, having sort read it from disk and write the results back to disk, and then reading them in; you can save a couple of steps by piping the data directly to sort using a piped open, and let it write the results to disk for you to read back. The following processes 10 million elements in around 3 minutes on my machine.

#! perl -slw
use strict;
use Benchmark::Timer;

our $N ||= 1e6;
my $T = new Benchmark::Timer;

my @array;
push @array, sprintf "Test%07d", int rand 32767 for 1 .. $N;

$T->start( 'sort -u' );
open my $pipe, "| u:sort -u > temp.tmp" or die $!;
print $pipe $_ for @array;
close $pipe;

open my $in, '<', 'temp.tmp' or die $!;
my $i = 0;
$array[ $i++ ] = $_ while <$in>;
close $in;
$#array = $i - 1; ## Corrected, with thanks to [johngg]

$T->stop( 'sort -u' );
$T->report;

printf "Array now contains %d uniq elements\n", scalar @array;
__END__
[ 8:17:47.37] c:\test>551753-2 -N=1e7
1 trial of sort -u (183.281s total)
Array now contains 32768 uniq elements
[download]

In theory, you could use IPC::Run3 to avoid hitting the disk at all, but then you have the problem of needing storage for both the input and output at the same time. The code above reads the data back into the original array and truncates the leftovers, so avoiding any extra memory growth.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re: Saving an array to a disk file Select or Download Code

Replies are listed 'Best First'.
Re^2: Saving an array to a disk file by johngg (Canon) on May 26, 2006 at 09:37 UTC
I might be barking up the wrong tree here but I think your `$#array = $i;` [download] should be `$#array = $i - 1;` [download] because the `$i` subscript has been post-incremented ready to receive the next value in the `... while <$in>;` loop. I don't think you can get 32768 unique values when generating them with `... int rand 32767 ...`. Cheers, JohnGG	[reply] [d/l] [select]
Re^3: Saving an array to a disk file by BrowserUk (Patriarch) on May 26, 2006 at 09:54 UTC
Indeed you are absolutely correct! Updated, thankyou++. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l]
Re^2: Saving an array to a disk file by Anonymous Monk on May 29, 2006 at 01:58 UTC
Dear BrowserUK, As I'm running in Linux, I enconter problem running your script above. On running it it gave this error `$perl save_disk_to_array_browseruk.pl sh: u:sort: command not found` [download] Then when I tried removing `u:` into `open my $pipe, "\| sort -u > temp.tmp" or die $!;` [download] It gave only: `Array now contains 1 uniq elements` [download] Is there any changes need to be made on your code above, if I'm running on Unix? Sorry to trouble you, I'm new in Perl. Thanks so much and hope to hear from you again.	[reply] [d/l] [select]
Re^3: Saving an array to a disk file by BrowserUk (Patriarch) on May 29, 2006 at 03:55 UTC
Yup. Switching from the win32 command to a unix command in order to run the code on unix is a good idea :) With that change made, the code should run correct on any system that has a `sort` command that takes a `-u` option. If that is not the case on your system, you will need to investigate the syntax for the command options on your machine, verify that the command is being run correctly, and generally work out why your system is different. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l] [select]