use strict; use warnings; my @lines1 = ( 'The, slow, lazy, brown and green dog', qq~I now want it to ignore special characters such as (), ",", ', +" etc.~, qq~I have code which finds duplicates in 2 text files~, ); my @lines2 = ( 'The slow lazy brown and green dog', qq~I now want it to ignore special characters such as (), ",", ', +" etc.~, qq~I have code which finds duplicates in 2 files and spits into 3 +files~, ); my %where; @lines1 = map {(my $clean = $_) =~ s/[^\s\w]//g; [$_, $clean]} @lines1 +; @lines2 = map {(my $clean = $_) =~ s/[^\s\w]//g; [$_, $clean]} @lines2 +; $where{$_->[1]} = ["1", $_->[0]] for @lines1; $where{$_->[1]} = [($where{$_->[1]}[0] .= "2"), $_->[0]] for @lines2; for (sort keys %where) { my $where = $where{$_}->[0]; my $what = $where{$_}->[1]; if ($where =~ /12/) { print "12: $what\n"; } elsif ($where =~ /1/) { print "1: $what\n"; } else { print "2: $what\n"; } }

Prints:

2: I have code which finds duplicates in 2 files and spits into 3 file +s 1: I have code which finds duplicates in 2 text files 12: I now want it to ignore special characters such as (), ",", ', " e +tc. 12: The slow lazy brown and green dog

is close, but only shows one variant of the dog line. It's really a question of how you actually want to handle that situation.


DWIM is Perl's answer to Gödel

In reply to Re: Finding duplicates in a text file by GrandFather
in thread Finding duplicates in a text file by ilottl

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.