Re: Better way to work with large TSV files?

Using a mass loader would indeed be the fastest option by far.

If you have to stick with Perl though, you can certainly gain speed. Tie::File has to do a lot of bookkeeping to a) find all the ends of lines b) update the file per your wishes. Maybe File::ReadBackwards is what you need: it is written to handle precisely the case you have efficiently.

Makeshifts last the longest.

Comment on Re: Better way to work with large TSV files?

Replies are listed 'Best First'.
Re^2: Better way to work with large TSV files? by BrowserUk (Patriarch) on Aug 20, 2004 at 17:35 UTC
How is File::ReadBackwards a substitute for Tie::File for this application? Examine what is said, not who speaks. "Efficiency is intelligent laziness." -David Dunham "Think for yourself!" - Abigail "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon	[reply] [d/l]
Re^3: Better way to work with large TSV files? by Aristotle (Chancellor) on Aug 20, 2004 at 18:02 UTC
`$sth->prepare( "INSERT INTO Blah ( a, b, c ) VALUES ( ?, ?, ? )" ); tie TSV, 'File::ReadBackwards', $filename; while( <TSV> ) { my @insert = processRow( $_ ); $sth->execute( @insert ); truncate $filename, tell TSV; } close TSV; unlink $filename unless $errors;` [download] Makeshifts last the longest.*	[reply] [d/l]
Re^4: Better way to work with large TSV files? by radiantmatrix (Parson) on Aug 24, 2004 at 13:39 UTC
I've tried this out, given the speed at which I was able to modify my existing code to work that way. The cpan://File::ReadBackwards module works tremendously well, with far less of a processor hit. I've noted a speed improvement of abour 500%! Thank you, Aristotle; I never would have found that module. ^{-- $me = rand($hacker{perl});}	[reply]