what have you tried so far?

and which of the records with duplicate first fields do you want to insert into the database? i don't quite understand why you're doing this, perhaps you need to rethink your database structure. as it stands, there are a few options.

i can help you with the pseudocode:

== as always, use strict and -w
== of course, use DBI for the database interface
== create an empty hash to store unique keys
== create an empty array for data to process
== for each data line: == | use split with'|' on line, store values in temp array
== | is first value from the array (temp_array[0]) in unique hash?
== | | yes : dump temp array and move on
== | | no : add first value to unique hash, add array to data_to_process array
== for each array in data_to_process array
== | add to database

this assumes you want to keep the first line with a repeated first field. if you want to keep the last line, the logic will have to change appropriately.

~Particle