I'm going to leave the filehandle method to others. I'd include category name unless you are positive that the same subcategory name is never used for two different categories.

You could also tie a hash to disk, using one DBM implementation or another depending on how you want to access your data (MLDBM is popular here).

But I think the best way would be to try using DBD::RAM. It seems to have been provided just for you! I tried it with your data for fun and it works great, letting you use SQL commands without a database server. Now I don't know how much memory this will want if you sort with "order by", and speed will probably be so-so but useable.

Of course if by huge you mean *huge* for raw read speeds you might want to pop over to mysql.com and get a powerful, free database server. In that case I would skip DBD::RAM and just parse the file and insert each line into the database, then run an SQL query on the resulting database table.

#!/usr/bin/perl -w

use strict;
use DBI;
use DBD::RAM;

my $dbh = DBI->connect('DBI:RAM:','usr','pwd',{RaiseError=>1});
$dbh->{f_dir} = '.'; # (default), or your path: '/path/to/files'
$dbh->func([
        [ 'items', 'PIPE',  'ramdata',  {col_names => 
          'id,category,subcat,code,description,picture' } ], 
       ],'catalog');

my $query = "select * from items where subcat='PEN' order by code";
my $sth = $dbh->prepare($query)|| die print $DBI::errstr;
$sth->execute || die print $DBI::errstr;

my @row;
while ( @row = $sth->fetchrow_array ) {
  print join(" ",@row),"\n";
  # or try this..   print "<img src=\"$row[5]\">\n";
}

# DBD::RAM lets you access a file with SQL.  PIPE separated
# is a built-in type which lets you read from one file and 
# write to another one without storing the whole thing in memory.
# Headings can come from the top of the file if you have a line
# containing them.
# You don't need a separate database server, just DBI, DBD::RAM,
# and SQL::Statement.  It can access remote files, and
# use different kinds of tables at the same time.
[download]

Good luck,

Matt

NEW, MORE: Check out this node. Some of the things mentioned above are illegal, like printing to a hash of filehandles without doing something sneaky (adding curly brackets and have Perl treat it as an object). Also you can't put such a hash into the diamond operator: $got = <$fd[0]> is illegal. Sorry for the braindead post yesterday. I researched and here are some ways to get filehandles to work in a hash.

I have not worked on using globs yet but this is another thing you could do; using Symbol::gensym is lightweight compared to IO::Handle, and you can store a glob in the hash instead of an IO::Handle object, so you could use the glob in readline where you would normally use the diamond operator. Info about globs (working code) would be useful if someone would like to add.

#!/usr/bin/perl -w

use strict;
use IO::File;

my %in;  my %out;
$in{b} = new IO::File; $out{b} = new IO::File;
$in{b}->open("<ramdata") || die $!;
$out{b}->open(">ramdata2") || die $!;
my $fi = $in{b}; my $fo = $out{b};

&v1;
# &v2; # this also works

close($in{b});  close($fo);

sub v1 {                     # works 
 while (<$fi>) { 
   print $fo "-> $_"; 
 }
}

sub v2 {                     # works
 my $x = $in{b};
 while (<$x>) { 
   print {$out{b}} "-> $_";  # sneak by print: treat as object
 }
}

sub nogoods {                # these doesn't compile
# while (<{$in{b}}>) {       # no obj allowed in diamond
# ... }
# while (readline($in{b})) { # bad: storing obj not fh glob
# ... }
}
[download]

Cheers,

Matt