in reply to Optimise database searching currently using DBD::CSV

#Load the 5000 rec file into a hash based on some unique
#field (like lastname + phone number).

foreach my $rec (@$client_recs) {
  $client{$rec->{lastname} . $rec->{phone}} = $rec;
}

#loop through each big file once
#and check the client hash for a match

foreach my $rec (@$big_file) {
  if($client{$rec->{lastname} . $rec->{phone}}) {
    #we've got a potential match!
    #now compare each individual field
    if($client{$rec->{lastname}.$rec->{phone}}->{firstname} eq $rec->{firstname} && ...) {
      #blah blah blah
    }
  }
}

#this way we're looping through the big files only once
#and doing a keyed in-memory search of the small file
#
#the other way around will loop through each big file
#5000 times.
  • Comment on Re: Optimise database searching currently using DBD::CSV

Replies are listed 'Best First'.
Re^2: Optimise database searching currently using DBD::CSV
by Anonymous Monk on Sep 06, 2004 at 15:08 UTC
    A good idea. I will let you know how I get in a weeks time