I see a couple of problems. Look here:

foreach $line (@lines) { my $id = $line =~ m/ID No: (\d+)/; if ($seen{$id}++){ # ...

You're going to get 1 for $id all the time because the match is in a scalar context instead of list context.

The other thing is you want to say "if ( ! $seen{$id}++ ) ...". The first time an $id is seen, $seen{$id}++ will be false. Every time after that, it will be true. Since you want it to be true once and false ever after, add the negation.

Here's the part I copied, rewritten:

foreach $line (@lines) { my ($id) = $line =~ m/ID No: (\d+)/; if ( ! $seen{$id}++){ # ...

I some other suggestions for you.

First and most important, check the value of open! You try to open your files, but if they fail, you'll never know. Also, it's a generally good idea to use lexical filehandles instead of the global ones you're using.

open my $in_fh, '<:utf8', 'input.txt' or die "Can't read 'input.txt': $!"; my @lines = <$in_fh>;

Second, you're doing everything in memory. If your input is huge, you could run out of memory. Instead of reading every line and then looping over them, consider using a loop that reads a line at a time instead. You'd also want to open your output files at the start and write into them during processing instead of collecting their eventual contents in (memory-based) arrays. Like this:

open my $in_fh, '<:utf8', 'input.txt' or die "Can't read 'input.txt': $!"; open my $purge_fh, '>>:utf8', 'purge.txt' or die "Can't append to 'purge.txt': $!"; open my $uniq_fh, '>:utf8', 'data.txt' or die "Can't write 'data.txt': $!"; my %seen; while ( my $line = <$in_fh> ) { my ($id) = ($line =~ /ID No: (\d+)/); if ( ! $seen{ $id }++ ) { print $uniq_fh, $line; $new_uniq++; } else { print $purge_fh, $line; } } close $in_fh or die "Close failed for input: $!"; close $purge_fh or die "Close failed for purge.txt: $!"; close $uniq_fh or die "Close failed for data.txt: $!";

Finally, if you don't already, Use strict and warnings!


In reply to Re: Removing duplicate lines based on a match by kyle
in thread Removing duplicate lines based on a match by carrerag

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.