Replacing all but one duplicate lines

johnirl has asked for the wisdom of the Perl Monks concerning the following question:

Hey Monks

quick question I have a script that reads in a file and removes a specific line. However what I want it to do is remove all occurances of the specific line except for the first one
Any ideas on how I could do this. I know this is a RDFM question but I am going on vacation today and I need to get it done quick

        open FILE, "<" . $filename or die "Couldn't open file " . $fil
+ename . " for reading";
        while (<FILE>){
                $temp .= $_ unless /$heading_to_print/;
        }
        close FILE;

        open FILE, ">" . $filename or die "Couldn't open file " . $fil
+ename . " for writing";
        print FILE $temp;
[download]

Thanks in advance quys

JohnIrl

Sum day soon I'Il lern how 2 spelI (r tYpe)

Comment on Replacing all but one duplicate lines Download Code

Replies are listed 'Best First'.
Re: Replacing all but one duplicate lines by demerphq (Chancellor) on Jul 19, 2002 at 12:41 UTC
`open INFILE, "<" . $filename or die "Couldn't open file " . $filename . " for reading : $!"; open OUTFILE, ">" . $filename or die "Couldn't open file " . $filename . " for writing : $!"; while (<INFILE>){ print OUTFILE $_; last if /$heading_to_print/; } while (<INFILE>){ print OUTFILE $_ unless /$heading_to_print/; } close INFILE; close OUTFILE;` [download] A little tip is in your error messages for open() and similar you shuld include the output of $!. Also personally I dont like using global filehandles. I prefer to use lexical ones, but this only works on later versions of Perl. HTH UPDATE This will remove any duplicate lines except the first, not just that of a specific line. (Although it might chew up a lot of memory if there are only a few dupes but many lines.) `open INFILE, "<" . $filename or die "Couldn't open file " . $filename . " for reading : $!"; open OUTFILE, ">" . $filename or die "Couldn't open file " . $filename . " for writing : $!"; my %hash; while (<INFILE>){ print OUTFILE $_ unless $hash{$_}++; } close INFILE; close OUTFILE;` [download] This could also be done as a one liner, `perl -ni.old -e "print $_ unless $_{$_}++" foo.txt` [download] Which really is quite nice isnt it. :-) (Note that the file will be "edited-in-place", and the original will be backed up as "foo.txt.old") Yves / DeMerphq --- Writing a good benchmark isnt as easy as it might look.	[reply] [d/l] [select]
Re: Replacing all but one duplicate lines by jmcnamara (Monsignor) on Jul 19, 2002 at 13:03 UTC
To do this you could modify your while loop as follows: `my $i = 0; while (<FILE>) { $temp .= $_ unless /$heading_to_print/ and $i++; }` [download] Your code suggests that you want to make the changes "in-place". You could do it all in a one-liner as follows: `perl -i.bak -ne 'print unless /pattern/ and $i++' file` Enjoy your vacation. `:-)` -- John.	[reply] [d/l] [select]
Re: Replacing all but one duplicate lines by Abigail-II (Bishop) on Jul 19, 2002 at 16:01 UTC
Any ideas on how I could do this. I know this is a RDFM question but I am going on vacation today and I need to get it done quick. Really? So, you spend time typing in your question in a tiny input field in a webpage, risking you are getting wrong, bogus and inefficient answers when a simple `perldoc -q duplicate` [download] would have given you the answer? Abigail	[reply] [d/l]
Re: Re: Replacing all but one duplicate lines by demerphq (Chancellor) on Jul 19, 2002 at 16:24 UTC
A big part of life is figuring out the correct questions to ask. Just because you have enough experience to do so doesnt mean we all do. I admit it, I almost _never_ use perldoc. Yves / DeMerphq --- Writing a good benchmark isnt as easy as it might look.	[reply]