use IO::File;

my $fh = new IO::File;
my %hash = ( file1=>$fh );
$hash{ file1 }->open(">subcat.txt") or die $!;
print $hash{ file1 } "Text here\a";
$hash{ file1 }->close();
[download]

   open inputfile
      read file into a hash, using the SubCatagory as the key
      (I dunno if you need ALL the data, but I`m sure you know
       how to stuff it in the hash at the SubCat Key (make a list 
       ref. or whatever you want.)
   close inputfile

   iterate over the keys in your hash (unique subcatagories)
      open Subcatagory file
       ...Do your thing on the file, using the data in your hash...
      close SubCatagory file
[download]

ChOas

~Particle

You could also tie a hash to disk, using one DBM implementation or another depending on how you want to access your data (MLDBM is popular here).

But I think the best way would be to try using DBD::RAM. It seems to have been provided just for you! I tried it with your data for fun and it works great, letting you use SQL commands without a database server. Now I don't know how much memory this will want if you sort with "order by", and speed will probably be so-so but useable.

Of course if by huge you mean *huge* for raw read speeds you might want to pop over to mysql.com and get a powerful, free database server. In that case I would skip DBD::RAM and just parse the file and insert each line into the database, then run an SQL query on the resulting database table.

#!/usr/bin/perl -w

use strict;
use DBI;
use DBD::RAM;

my $dbh = DBI->connect('DBI:RAM:','usr','pwd',{RaiseError=>1});
$dbh->{f_dir} = '.'; # (default), or your path: '/path/to/files'
$dbh->func([
        [ 'items', 'PIPE',  'ramdata',  {col_names => 
          'id,category,subcat,code,description,picture' } ], 
       ],'catalog');

my $query = "select * from items where subcat='PEN' order by code";
my $sth = $dbh->prepare($query)|| die print $DBI::errstr;
$sth->execute || die print $DBI::errstr;

my @row;
while ( @row = $sth->fetchrow_array ) {
  print join(" ",@row),"\n";
  # or try this..   print "<img src=\"$row[5]\">\n";
}

# DBD::RAM lets you access a file with SQL.  PIPE separated
# is a built-in type which lets you read from one file and 
# write to another one without storing the whole thing in memory.
# Headings can come from the top of the file if you have a line
# containing them.
# You don't need a separate database server, just DBI, DBD::RAM,
# and SQL::Statement.  It can access remote files, and
# use different kinds of tables at the same time.
[download]

Matt

NEW, MORE: Check out this node. Some of the things mentioned above are illegal, like printing to a hash of filehandles without doing something sneaky (adding curly brackets and have Perl treat it as an object). Also you can't put such a hash into the diamond operator: $got = <$fd[0]> is illegal. Sorry for the braindead post yesterday. I researched and here are some ways to get filehandles to work in a hash.

I have not worked on using globs yet but this is another thing you could do; using Symbol::gensym is lightweight compared to IO::Handle, and you can store a glob in the hash instead of an IO::Handle object, so you could use the glob in readline where you would normally use the diamond operator. Info about globs (working code) would be useful if someone would like to add.

#!/usr/bin/perl -w

use strict;
use IO::File;

my %in;  my %out;
$in{b} = new IO::File; $out{b} = new IO::File;
$in{b}->open("<ramdata") || die $!;
$out{b}->open(">ramdata2") || die $!;
my $fi = $in{b}; my $fo = $out{b};

&v1;
# &v2; # this also works

close($in{b});  close($fo);

sub v1 {                     # works 
 while (<$fi>) { 
   print $fo "-> $_"; 
 }
}

sub v2 {                     # works
 my $x = $in{b};
 while (<$x>) { 
   print {$out{b}} "-> $_";  # sneak by print: treat as object
 }
}

sub nogoods {                # these doesn't compile
# while (<{$in{b}}>) {       # no obj allowed in diamond
# ... }
# while (readline($in{b})) { # bad: storing obj not fh glob
# ... }
}
[download]

Matt

I like the autovivification in 5.6.0:

#!/usr/bin/perl

use strict;
use warnings;

{
    my %file_handle;
    while (<DATA>) {
        my ($id, $category, $sub_category, $code, $description, $pictu
+re) =
             split /\|/;
        # massage data into a printable string

        $file_handle{$sub_category} ||= get_handle($sub_category);

        print {$file_handle{$sub_category}} "something\n";
    }
}   # As %file_handle goes out of scope
    #   all file handles close implicitly

sub get_handle {
    my $file_name = shift;
    open my $fh, '>', $file_name or die "Cannot write to $file_name: $
+!";
    return $fh;
}


__END__
0|OFFICE EQUIPPMENT|PEN|PDS01|an ordinary pen|pen.gif
1|OFFICE EQUIPPMENT|PEN|PDS02|another pen|pen2.gif
2|OFFICE EQUIPPMENT|PAPER|PA003|white sheets to write on|paper.gif
3|OFFICE EQUIPPMENT|PEN|PDD50|the greatest pen|pen50.gif
4|OFFICE EQUIPPMENT|PEN|PDS01|an ordinary pen|pen.gif
[download]

Warning! code not verified and is only a concept

my @stock;
open FILE, "stockfile";
while(<FILE>) {
   push @stock,$_;
}
close FILE;

while(@stock) {
 foreach $line (@stock) {
    @items = split ´|´, $line;
    open $item[2], ">>some_filename";
       put_items_into_it;
    close $item[2]
    delete_that_line_from_stock;
 }
}
[download]

while(<FILE>) {
   push @stock,$_;
}

# Would be better as...
@stock = <FILE>;
[download]

Why wrap a while around a foreach here?

while(@stock) {
 foreach $line (@stock) {
[download]

A straight foreach will do just as well. Of course in that case there is no need for @stock at all...

while (my $line = <FILE>) {
    # do stuff
}
[download]

Additionally this will not be memory intensive because it will only store one line in memory at a time. If you are attempting to keep only one filehandle open at a time via...

open $item[2], ">>some_filename";
       put_items_into_it;
    close $item[2]
[download]

You might be better off using checking if the next item to be written out goes into the same FILE that is already opened. No reason to close it and open it again immediately.

my $cat = undef;
while (my $line = <FILE>) {
    @items = split('|', $line);
    if ($cat ne $items[2]) {
        close OUTFILE if (defined $cat);
        open OUTFILE, ">>some_filename";
        $cat = $items[2];
    } 
    print OUTFILE ## Whatever ###;
}
close OUTFILE;
[download]

delete_that_line_from_stock;
[download]

Is not neccessary, since there is no @stock anymore.

my $fh = IO::File->new(">$filename");
print $fh "Hello\n";
close($fh) or die "Can't close\n";
[download]

--- So Fast, So Numb

I'd store all the file handles in a hash, indexed by subcategory. When all is over, you then can close them, foreach value in the hash.

my %fh; # hash of filehandles
while ( <> ) {
    # do something to find out the $subcat(egory)
    # and open the file if not done already
    $fh{$subcat} || open($fh{$subcat}, ">$pathtofile"
        or die "Aaargh: $!";
    # now you can print stuff to the file
    print { $fh{$subcat} } @stuff;
}
# Close down everything
close $_ foreach values %fh;
[download]

--bwana147