in reply to DBI insert statement

Looking at the examples it seems the structures of the tables is the same, you are inserting a subset to the new table. If so and there isn't some other parsing happening can't you offload the whole to the database

insert into targettable select * from sourcetable where ...

If target doesn't need to permanently exist you can

create targettable as (select * from sourcetable where...)

Then drop the new table at end of run.

Replies are listed 'Best First'.
Re^2: DBI insert statement
by soonix (Canon) on Nov 22, 2016 at 19:13 UTC
    OP states they want to insert the data into another DB, so a pure SQL solution is possible only if both databases are serviced by the same server (and are more or less just different namespaces).

    Update: Just to make it clear: I concur with tweetiepooh that in setups, where it is possible at all, you can bet important body parts on it being the most efficient solution, too.

      Don't know enough about their setup but Oracle allows DB links between databases, even on different servers and the idea would still work, you just need to tell the SQL that one database is over a link. This still offloads the work to the database, maybe even better if both servers are in a data centre as traffic doesn't flow from one DB to Perl to other DB (OK Perl maybe at same place as one of the DB's).

      Again with Oracle you can use views and synonyms to make the remote table appear to be local so you only need to connect to one database. When I was doing something similar I had to create the link on the fly because I was dealing with multiple remote databases and wanted to use the same link name so that is also possible. This also meant that when script wasn't running the link was absent too so users couldn't access the remote databases.

        And other systems have similar things:

        PostgreSQL has postgres_fdw and other fdw's ( foreign data wrappers ) to connect to data sources outside the current postgres server.