in reply to DBI - need for speed

If a large percentage of the 8000 or so records in SRCFILE are going to end up as new data to be inserted into the table, then you will save A LOT of time by moving those records into a separate plain text file that can be "bulk-loaded" into the table by whatever means is provided in your particulat DBMS (e.g. "sqlload" in Oracle, "mysqlimport" or some such). Single-record-at-a-time inserts with DBI are painfully slow compared to a native DBMS utility that is already optimized for inserting/appending large quantities of data into tables.

Follow the advice given in the first reply -- do one query at the outset to find out what records are already in the table -- then open SRCFILE for input, open some other file for output, and for each line of input, write a suitable data line to output if the corresponding record needs to be added to the table. After output file is fully written, run the bulk-loader utility of choice for DBMS to add that data to the table (this can be done from within the same perl script that writes the file).