in reply to Re^2: Loading file into memory
in thread Loading file into memory
You want to process the CSV file and put into a database or some sort. Either a DB file or SQLite. If you use SQLite, make sure you index the fields you'll be searching against. If you use a DB file, (such as the Berkeley DB) you'll want to think carefully about your data structure. MLDBM is also worth looking at. Your query times will improve dramatically.
So your flow should look something like this:
my $dbh = Read_Huge_File_Into_DB( $huge_file_path ); my @customers = Process_Customer_Information_File( $dbh, $file_path ); Print_Report(\@customers); sub Process_Customer_Information_File { my $dbh = shift; my $file = shift; open( $info, '<', $file ) or die "Uh oh $!"; my @customers_found; while ( my $line = <$info> ) { my $customer_data = ParseCustomerData($line); my $name = $customer_data->[NAME]; if ( Customer_Found( $dbh, $name ) ) { push @customers_found, $customer_data; } } return @customers_found; }
If it were me, I'd use SQLite.
TGI says moo
|
|---|