PapaMonzano has asked for the wisdom of the Perl Monks concerning the following question:
running activestate perl 5.6.1 build 631 on win2k.
I have a 7.5 gig file from a database with 40,000,000 or so records (1 per line). I need to check each record in the file (using a regex) and then write the validated/corrected data out to files of 1,000,000 records each. I am reading the source file one line at a time (while(<SOURCE>){etc...};), processing the record and then appending it to the output file. after i print the 1,000,000th record i close the destination handle and open a new file and continue to print to it, etc... but when i get to the 10,546,603rd record (about 2 gigs into it) the program exits. Even if i just count the lines, the same thing happens. the last line it reads is a truncated record and no \n. I'm sure that isn't the last line in the file, and I was able to successfully import all of the records back into MS SQL server.
I suspect your perl isn't compiled with the uselargefiles option - it is needed for files bigger than 2 gigs..
you can test this with perl -V Check for the line starting with config_args it should have uselargefiles=define in there if you have large file support...
--
Joost downtime n. The period during which a system
is error-free and immune from user input.