I am not sure what you mean by "a single query with 3300 rows." I mean $sth->execute() is called for each row, and a $dbh->commit() is called once every 100_000 rows.
As I explained earlier, for many database, there's can be significant difference between creating a insert statement that inserts a single row, and calling execute() for each row, or a single statement that inserts thousands† of rows in one shot. Apparently, you haven't tried out that variant.

As for the other questions, they are important if you need maximum speed. Giving me the answers is useless; you'd have to consider it all. They are important to determine where you bottleneck is. Disks? The channels to your disks? Memory? CPU? But this all goes way beyond a web based Perl forum.

If it is rewriting the entire table everytime then it is a very silly db
Says you. There are storage engines that keep rows in primary key order. That's not silly. This makes searching on primary keys fast, and getting ranges fast as well. It entirely depends on the application how one wants a database tuned. It often makes sense to tune a database for fast retrieval, even if it makes inserts slower. Of course, 'rewriting' doesn't mean it's writing the entire table for each row. A page will get reordered. A full page may be split into two.
How "hot" is the table? Don't know what that means.
How much of the table is cached somewhere? (In-process pages, filesystem buffers (assuming you have the additional filesystem overhead), disk controller caches, etc).
What's the physical layout of where the data is written to? Don't know what that means.
Are you writing to a single disk? Multiple disks? SAN? Mirrorred disks? RAID? What RAID? Are you striping? How's your data storage connected? Copper? Fiber channel? If you have a 10 TB database, this matters. A lot. Specially since you've insignificant memory compared to size of the database. You may have to try out several setups to determine what works for you. You said diskspace is cheap. Sure, raw diskspace is cheap. Getting a 10 TB database to perform well may not be so cheap.

There's a lot more to database that just having lots of disk space and a bunch of SQL statements.

†The more the better. How many rows you can fit in a single statement is usually delimited by the maximum package size.

In reply to Re^5: Speeding up Postgres INSERTs by JavaFan
in thread Speeding up Postgres INSERTs by punkish

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.