in reply to Reading 39GB Flat File?

Your database (which one you don't mention) probably has a way to import bulk data. For postgres, there is the COPY command which will read from STDIN (or directly from file):

cat file.txt | psql -d dbname -c "copy mytable from stdin csv"

'mytable' must have the appropriate columns, of course.

( A site-search on 'Bulk import' will probably yield bulk import methods for each RDBMS: bcp for sybase/mssql, imp for oracle, etc. )

update:

ok, ok... To appease the venerable and dodge the infamous award, this should be written as:

< file.txt psql -d dbname -c "copy mytable from stdin csv"

Although a process isn't as expensive as it used to be :)

Replies are listed 'Best First'.
Re^2: Reading 39GB Flat File?
by merlyn (Sage) on Nov 12, 2009 at 17:13 UTC
    cat file.txt | ...
    Had this been a usenet posting, I would have given you a Useless use of cat award.

    -- Randal L. Schwartz, Perl hacker

    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.