Basically I read in the contents of a very large file as an array and then bucket-hash it using the Data::Bucket index() method:

=head2 index

 Usage     : my $bucket = Data::Bucket->index(data => strings, %other)
+;
 Purpose   : Build a data structure with @strings partitioned into buc
+kets
 Returns   : An object.
 Argument  : A list of data compatible with the compute_index() functi
+on

=cut

sub index
{
    my ($class, %parm) = @_;

    exists $parm{data} or die "Data must be passed for indexing" ;
    ref $parm{data} eq 'ARRAY' or die 'You must pass an array ref';

    my $self = bless (\%parm, ref ($class) || $class);

    $self->bucket_hash;

    return $self;
}

=head2 bucket_hash

 Usage     : Called internally by index()
 Purpose   : Partition $self->{data} by repeated calls 
   to $self->compute_record_index
 Returns   : Nothing
 Argument  : None.

=cut

sub bucket_hash
{
    my ($self) = @_;

    for my $data (@{$self->{data}}) {
    my $index = $self->compute_record_index($data);

    my @index = ref $index eq 'ARRAY' ? @$index : ($index) ;
    for (@index) {
        push @{ $self->{bucket}{$_} } , $data ;
    }
    }

    return $self;
}
[download]

1. rewrite the bucket_hash method used by index() so that it writes to a SQLite database. This approach allows for in-memory or on-disk databases as needed. I'm still kicking myself for not doing this in the first place, but I figured I would never run out of memory.
2. Some sort of method of tie a perl hashref to a disk. I think there are some modules for saving them to sleepycat db files or something... any ideas here? This would save me from rewriting my code.
3. Increasing the virtual memory on my machine... perhaps I can just jack up the virtual memory ... but would Cygwin Perl know how to take advantage of such memory or can my hashref only occupy main physical RAM?

[tag://cpan-bucket-hash,memory,sqlite]
[download]