in reply to Is there a more efficient way?

IomSpace:

If speed is the issue, I'd use this approach:

  1. Create a surrogate temporary table with no indexes for each table you're loading.

  2. Bulk load your temporary tables. (I don't know MySQL, so I'm unfamiliar with its bulk data loader.)
  3. Insert new records from your temporary tables into your production tables.

    -- Only insert them if they don't exist INSERT INTO dev.blocks (names) SELECT names FROM dev.tmp_blocks WHERE names NOT IN ( SELECT names from dev.blocks ) INSERT INTO dev.vials (block_id, lt, well_position, gene, barcode) SELECT block_id, lt, well_position, gene, barcode FROM dev.tmp_vials WHERE ... clause to prevent inserting duplicates ...
  4. Drop your surrogate tables.

This way, you can let the database do the work as efficiently as possible. You'll reduce the number of transactions on the database, reduce the traffic between your server and the database server, reduce the I/O consumed by the database, etc.

...roboticus