Remove lines that contain matching values from csv.

urbs33 has asked for the wisdom of the Perl Monks concerning the following question:

Perl Monks,

I am looking for some input on filtering a csv to unique records based on a specified field. I have a list of records that may exist on multiple servers. I am generating a list of the files, and some additional information about them, including the server that they reside on. I only want to output one record of the file, even if it is one multiple servers. Let's suppose that the field to match is field one. I will have something like this.,

unique_id,server_name,modification_date,size,another_field,another_field,

There are thousands of lines in this file. If one fo the files from this report is copied to another server, the unique_names will match, but the server name will not. I only want one record of each unique_name considered int his report and do not care from which server name. This is a unix OS, so if there is an easier way to do it with awk, sort|uniq, or other native commands, I am open to that. I'm just kinda stumped since the rest of the line will not match exactly.

Thanks!!

Comment on Remove lines that contain matching values from csv.

Replies are listed 'Best First'.
Re: Remove lines that contain matching values from csv. by Anonymous Monk on Oct 08, 2012 at 16:01 UTC
CSV manipulation is the most popular of FAQs everywhere. An answer might use Text::CSV or Text::xSV Examples at Merge two files with similar column entries Opening multiple files ( csvpaste.pl ) Open multiple file handles? X, Y Table structure How to extract the particular residues from PDB files(text csv split hash bioperl) Select only desired features from a text (text csv split hash bioperl) Sort on Table headers (text csv split hash bioperl) I need help joining tab-delimited files/tables! split then join based on common value in field Smart way to read a file vertically? reformatting tab delimited file edit a CSV and "in-place" replacement Re: Reversing a mysql table, transpose-tsv -- invert a tab-delimited table	[reply]
Re: Remove lines that contain matching values from csv. by BrowserUk (Patriarch) on Oct 08, 2012 at 16:09 UTC
`perl -F, -anle"++$uniq{$F[0]} == 1 and print" infile > outfile` [download] With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP Neil Armstrong	[reply] [d/l]
Re: Remove lines that contain matching values from csv. by fluffyvoidwarrior (Monk) on Oct 09, 2012 at 15:52 UTC
Perhaps I'm missing the point of your question but..... If the fields are ordered as you seem to suggest and the unique id is in position 1 (so you can short circuit for speed - otherwise you'd have to regex a whole line - slower than anchoring at start or substr) can't you just treat the csv files as text files, ie a bunch of arrays. Parse each one and compare position 1 (the id field) with a cumulative output array for uniqueness. So long as the output is less than about 100,000 not-huge lines Perl should do this in a few seconds per input file (if you've optimised your code). For obvious reasons simple textfile handling is a lot faster than using CSV libraries) I wouldn't know how to do this with a one-liner but I don't see why you would have to. Maybe it's not the neatest of solutions but it is a guaranteed, self contained solution for half hours work, that other people will easily understand in the future.	[reply]