If you have a database, then why don't you let the database do the work? → Select DISTINCT
(the link is for SQLite, but as this is fairly vanilla SQL, the other candidates should understand it more or less unmodified, even DBD::CSV)