Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks I have an dictionary of index and the corresponding terms:
1 eat 2 habit 3 boy 4 man-kind 5 man 6 kind ..
I want to search in my text and replace each term with its index. I have my code and it works but I have two problems:

1- when in my text I see "Eat" or "eat" I want to replace them with 1. my index is all lowercase but my text has lower and uppercase. so if my text is "XXX EAT Eat eat xxx" > I want it "XXX 1 1 1 xxx" without lowercasing the original text.

2- When I reach the terms such as "man-kind" , it would not be replaced by 4. thats the problem with "-".

this is my code:
my %dict; while ( <DATA> ) { my ( $key, $val ) = /^(\d+)\s+(\w+)/; $dict{ $val } = $key; } my $cc = join '', keys %dict; my ( $min ) = my ( $max ) = map length, keys %dict; for ( map length, keys %dict ) { $min = $_ if $min > $_; $max = $_ if $max < $_; } my $pattern = qr/\b([$cc]{$min,$max})\b/; while (my $line = <INFILE>) { $line =~ s/(\S+)/$dict{$1} || $1/eg; print $line; }
Where INFILE is my text and DATA is my index dictionary. Thanks in advance.

Replies are listed 'Best First'.
Re: seaching for replacement
by kennethk (Abbot) on Nov 13, 2009 at 16:40 UTC
    As a note for the future, when posting code, it's very helpful to also provide the input to that code - in this case your DATA block and INFILE. See How do I post a question effectively?.

    If you want to do your replacement in a case insensitive fashion, you can use case insensitive matching (i modifier, see perlretut) and run your matched variation through the lc function. You can also join your keys with an alternator to generate a regular expression that will exactly match only your keys. Something like:

    #!/usr/bin/perl use strict; use warnings; my %dict; while ( <DATA> ) { my ( $key, $val ) = /^(\d+)\s+(\S+)/; $dict{ $val } = $key; } my $regex = '\b' . join('\b|\b', map "\Q$_\E", sort {length $b <=> len +gth $a} keys %dict) . '\b'; open my $fh, '<', 'junk.txt' or die "Open failed: $!"; while (defined(my $line = <$fh>)) { $line =~ s/($regex)/$dict{lc($1)}/ieg; print $line; } __DATA__ 1 eat 2 habit 3 boy 4 man-kind 5 man 6 kind

    tested with input file junk.txt:

    The boy has a habit of eating kind of like a man. Man-kind, as a grou +p, does not wear habits. Kind of you to watch what you eat.

    which gives the output:

    The 3 has a 2 of eating 6 of like a 5. 4, as a group, does not wear habits. 6 of you to watch what you 1.

    Note the use of the quote-meta operators \Q and \E to avoid some nasty corner cases. Also note that your original dictionary read was not reading in man-kind because \w does not match -. I've fixed that by correcting the regular expression and sorting the keys from longest to shortest in the join, so hyphenates are guaranteed to match before any of their sub strings. I've added a defined test to your file read, since as you had it your while-loop would have exited if your input file has a blank line (defined is implicit in the while(<$fh>) construct). I've also changed to a tested, 3-argument open and indirect file handle, all of which are considered Good Things(tm).

Re: seaching for replacement
by halfcountplus (Hermit) on Nov 13, 2009 at 16:34 UTC
    The case sensitivity issue is simple, use "i" with your s/ to make the operation "case insensitive":
    $line =~ s/(\S+)/$dict{$1} || $1/egi;
    The problem with "man-kind" is this: \S includes "-", but \w does not!
    my ( $key, $val ) = /^(\d+)\s+(\w+)/;
    If you print the value of your keys, I think you will find #4 is just "man".

    By the way, the "key" to a hash is it's name. The "value" of a hash is the data it contains. Altho it will not affect the functioning of your code, you have this nomenclature backward in the code which creates the dictionary.
Re: seaching for replacement
by johngg (Canon) on Nov 13, 2009 at 19:07 UTC

    I wondered how you might cope with dictionary words with a space in them, like "bar graph". The solution I came up with was to do a descending lexical sort of the dictionary keys when constructing the regex. That way "bar graph" will be matched in preference to "bar" and "graph" will already have been consumed by the regex if preceded by "bar". This code

    use strict; use warnings; use Data::Dumper; open my $dictFH, q{<}, \ <<'DICT' or die qq{open: < HEREDOC: $!\n}; 1 eat 2 habit 3 boy 4 man-kind 5 man 6 kind 7 bar 8 graph 9 bar graph DICT my %dict = reverse map { split m{\s+}, $_, 2 } map { chomp; $_ } <$dictFH>; close $dictFH or die qq{close: < HEREDOC: $!\n}; print Data::Dumper->Dumpxs( [ \ %dict ], [ qw{ *dict } ] ); my $text = <<'TEXT'; The boy has a habit of eating kind of like a man. Man-kind, as a group, does not wear habits. Kind of you to watch what you eat. Make a bar graph of what drinks are drunk in a bar and the graph could be coloured in. TEXT my $rxDict = do { local $" = q{|}; qr {(?xi) \b( @{ [ map quotemeta, sort { $b cmp $a } keys %dict ] } )\b } }; $text =~ s{$rxDict}{ $dict{ lc $1 } }eg; print $text;

    produces

    %dict = ( 'eat' => '1', 'man' => '5', 'kind' => '6', 'bar' => '7', 'man-kind' => '4', 'bar graph' => '9', 'boy' => '3', 'graph' => '8', 'habit' => '2' ); The 3 has a 2 of eating 6 of like a 5. 4, as a group, does not wear habits. 6 of you to watch what you 1. Make a 9 of what drinks are drunk in a 7 and the 8 could be coloured in.

    I hope this is of interest.

    Cheers,

    JohnGG

    Update: Sorting the keys in ascending lexical order shows what happens when the shorter is chosen before the longer. The transformed text becomes

    The 3 has a 2 of eating 6 of like a 5. 5-6, as a group, does not wear habits. 6 of you to watch what you 1. Make a 7 8 of what drinks are drunk in a 7 and the 8 could be coloured in.
Re: seaching for replacement
by toolic (Bishop) on Nov 13, 2009 at 16:42 UTC
    To resolve the case-sensitive issue, try this:
    while (my $line = <INFILE>) { my @words; while ($line =~ /\S+/g) { push @words, lc $1; } for my $word (@words) { if (exists $dict{$word}) { $line =~ s/$word/$dict{$word}/gi } } print $line; }

    Update: To resolve your dash issue, you could use a character class and quotemeta as follows:

    my ( $key, $val ) = /^(\d+)\s+([\w-]+)/; ... my $pattern = quotemeta "\b([$cc]{$min,$max})\b/";

    Update: Fixed 1st regex thanks to AnomalousMonk