in reply to processing huge files

You can use Tie::File to read the file as an array and then parse and insert into database. 
use Tie::File;
tie @array, 'Tie::File', $filename or die $!;

for (@array) {
   # parse
   # insert
}

[download]
Igor.

Igor S. Lopes - izut
surrender to perl. your code, your rules.

Replies are listed 'Best First'.
Re^2: processing huge files
by geektron (Curate) on Aug 02, 2005 at 16:55 UTC
    after reading the perldoc for Tie::File it sounded like a good idea™ ... but it's not behaving per docs.

    I'm trying: 

    tie my @array, 'Tie::File', $filename, mode => O_RDONLY;
my $recs = @array;
warn "we have $recs records to parse ";
pop @array;

foreach my $element ( @array ) {
   print "ok, we'd be updating $element \n";
}

    [download]
    the  tie statement is straight out of the perldoc.

    i need to skip the first line during processing since it's a "header" line.

    the error that comes back is: 

    we have 280579 records to parse  at ./upd_res_data2.pl line 72, <FH> l
+ine 280579.
Couldn't write record: Bad file descriptor at /usr/lib/perl5/5.8.0/Tie
+/File.pm line 665, <FH> line 280580.

    [download]
    and i don't think it *should* be writing, because it's (supposedly) opened read-only. any insight?
[reply]
[d/l]
[select]
      I tried:

      file.txt 

      # header
1
2
3
4
5
6

      [download]
      test.pl 
      use Tie::File;
tie @f, 'Tie::File', "file.txt" or die $!;
foreach (@f[1..$#f]) {
    print $_, "\n";
}

      [download]
      Result is: 
      1
2
3
4
5
6

      [download]

      Igor S. Lopes - izut
      surrender to perl. your code, your rules.
[reply]
[d/l]
[select]
        yes, that works fine, but if you try to add the  mode option, at least for me it ignored the RO nature and threw the error.
[reply]
[d/l]

In Section Seekers of Perl Wisdom