“Why sort your keys externally when you can get the DB to do it for you?”

Indeed...

To seek to answer that question, I give you today's workload of 11,344,209 telephone call records. You have exactly 4 hours wall-time to process them. If you imagine that you have enough time to put all those records into a B-tree-indexed random file, I have a bridge to sell you. Instead, to solve this problem and to do so consistently on a daily basis, it will be necessary for you to accomplish the same workload -- with utter reliability and consistency -- much faster.

It may come as an utter and complete shock to you to fathom that your grandfathers, armed with nothing more punched-card tabulators and sorters, with nary a digital computer in sight, could do that. Every day. Under wartime conditions.

#!/usr/bin/perl

use strict;
use warnings;

use BerkeleyDB;

my $db_file = '/home/snowhare/perl_monks/example.db';
unlink $db_file;

my $db    = BerkeleyDB::Btree->new( -Filename  => $db_file,
                                    -Cachesize => 700_000_000,
                                    -Flags     => DB_CREATE,
                             );  
srand;
for(my $count = 0; $count < 12_000_000; $count++) {
        my $random_value = rand(16776216);
        my $status = $db->db_put( "$random_value" => "$count" );
}
undef $db;


[snowhare@blue-bay perl_monks]$ time ./big_btree.pl 

real    3m47.121s
user    3m7.978s
sys     0m4.884s
[download]

This was on a desktop class machine with 1.5GBytes of RAM and an AMD Athlon 64 3000+ processor running Fedora Core 6 Linux.