Now try it on a file with a million lines...
Update: Apparently, this is seen as sarcasm or otherwise unhelpful; so here are some statistics:
Using Tie:File and an inplace Fischer-Yates shuffle to sort various sized files:
- 100 lines: 58 milliseconds.
- 1,000 lines: 2 seconds
- 10,000 lines: 194 seconds
- 100,000 lines: after 3 1/2 hours of cpu I got sick of listening to the fan thrashing itself to death trying to keep the cpu cool, and aborted.
- 20, 000,000 (the OP's task): Probably best measured in half-lives of Plutonium.
The test code should anyone wish to verify my figures.
#! perl -slw
use strict;
use Tie::File;
use Benchmark::Timer;
our $N ||= 1000;
sub shuffle {
my $ref = @_ == 1 ? $_[ 0 ] : [ @_ ];
for( 0 .. $#$ref ) {
my $p = $_ + rand( @{ $ref } - $_ );
@{ $ref }[ $_, $p ] = @{ $ref }[ $p, $_ ];
}
return unless defined wantarray;
return wantarray ? @{ $ref } : $ref;
}
open OUT, '>', 'junk.dat' or die $!;
printf OUT "%030d\n", $_ for 0 .. $N;
close OUT;
my @lines;
tie @lines, 'Tie::File', 'junk.dat';
my $T = new Benchmark::Timer;
$T->start( "shuffle $N" );
shuffle \@lines;
$T->stop( "shuffle $N" );
$T->report;
Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
"Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
| |
For: |
|
Use: |
| & | | & |
| < | | < |
| > | | > |
| [ | | [ |
| ] | | ] |
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.