in reply to Re^2: Best method to load a csv file.
in thread Best method to load a csv file.

The database loading portion of the task is often resource intensive enough that you will want to use your database's import facility. Just process your data in your script prior to giving it to the database. That's the way most of us do that anyway. You should also be able to reduce your runtime from a few hours per file to a few minutes for everything if you do this.

Replies are listed 'Best First'.
Re^4: Best method to load a csv file.
by BravoTwoZero (Scribe) on Dec 17, 2004 at 16:02 UTC

    Plus, you can always call your database import utility from the same perl program that modifies your data when it's complete. With that number of records, I'd still write it out to a text file, though, before calling the external db load utility.

    Amatuers discuss tactics. Professionals discuss logistics. And... my cat's breath smells like cat food.
[reply]
      That's true. So back to the question of parsing the csv file, which do you think is better or faster? The way I am doing it now with Text::ParseWords or DBI::CSV or Text::csv?

      Thanks,
      Sam
[reply]
        Use Text::xSV or Text::CSV_XS. The other modules tend to have subtle bugs in them.

        My question is why aren't you loading the data into temp tables, then using PL/SQL to clean the data up later? That's what I did when I had to do something similar on a daily basis and it was quite fast ...

        Being right, does not endow the right to be rude; politeness costs nothing.
        Being unknowing, is not the same as being stupid.
        Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
        Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

[reply]

In Section Seekers of Perl Wisdom