Re: DBI Question: bulk insertion with Unique-key constraint

Some random thoughts:

This isn't a Perl-specific question - it's a generic question about massaging data before inserting to a datastore.
If speed is an issue, then the bulk-loader is the right way to do it.
Determining uniqueness is a problem for a hash. Or, in your case, a hash-of-hashes-of-hashes (HoHoH)
Auto-incrementing a PK is a matter of inserting NULL. Read the documentation for your bulk-loader.
In fact, read the documentation for all the pieces you're working with. Most of the questions you have answered in the FAQ for each.

My criteria for good software:

Does it work?
Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

Comment on Re: DBI Question: bulk insertion with Unique-key constraint

Replies are listed 'Best First'.
Re^2: DBI Question: bulk insertion with Unique-key constraint by lihao (Monk) on Apr 24, 2008 at 20:23 UTC
I know this is not a Perl-specific question. Just I want to find a nice Perl solution if there is any.. Many thanks..	[reply]
Re^3: DBI Question: bulk insertion with Unique-key constraint by dragonchild (Archbishop) on Apr 24, 2008 at 20:32 UTC
In other words, you saw the first of 5 points and completely ignored the rest of what I said. Inserting records through DBI will always be slower than the bulk-loader provided with the RDBMS. Often, 100x slower. Therefore, the proper solution is to prepare a file for the bulk-loader. You were worried about auto-incrementing primary keys. The solution in MySQL is to insert a NULL. In Oracle, the solution is different. Sybase has a third solution. You were worried about unique keys. Bulk-loaders provide a mechanism for solving this problem. Alternately, you should use a hash because that's the way to unique-ify a list of values. Open your head! My criteria for good software: Does it work? Can someone else come in, make a change, and be reasonably certain no bugs were introduced?	[reply]