Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Fellow Monks,

what would be there preferred method for parsing through large text files (70+ MB each). I have a number of these to parse, they are tab delimited, and I need to parse through them line by line and insert the info into a database.
Would Tie::File be useful in this situation?

Thanks a lot,
Jonathan

Replies are listed 'Best First'.
Re: parsing large text files
by dragonchild (Archbishop) on Dec 02, 2004 at 14:21 UTC
    Use whatever utilities are provided by your database. For example, Oracle and MySQL both have excellent loader utilities that will run at least 1000 times faster than anything Perl can do.

    If you insist on using Perl, I would use Text::xSV to parse the files.

    Being right, does not endow the right to be rude; politeness costs nothing.
    Being unknowing, is not the same as being stupid.
    Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
    Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

Re: parsing large text files
by Eimi Metamorphoumai (Deacon) on Dec 02, 2004 at 14:21 UTC
    You could use Tie::File, but it sounds from your description as if you really don't need to. If each line is independant of all others, then you can just loop through one at a time. Just make sure you don't store anything that will accumulate over time. Just use a simple
    while (<>){ my @fields = split /\t/; #insert them into the database }
    If there's more going on, then you'll have to tell us what's complicating it so we can figure out what else to suggest.