I couldn't find the stuff I had done before, probably was pretty sad Perl since I wrote it back '96. Here, however is some stuff to start with that I hacked just now. (I'll continue with this, but probably in my own direction.) The thesaurus file is merely a list of keywords followed by synonyms. The newWords should probably be appended to the thesaurus file with an appropriate # comment for each word.

A thought, it might be possible to get some first take synonym solutions by parsing the returned text from a dictionary site. 

#!/usr/bin/perl -w
use strict;
use warnings;

my $thesaurusPath = "";
my $thesaurusFile = "thesaurus.nav";
my $ignore = '//'; # key to ignore in the thesaurus file
my $thesaurus = {};
my $newWords = {};

# read in the thesaurus word list
open CMDS, "<${thesaurusPath}${thesaurusFile}" or die
    "cannot open \"$thesaurusFile\" for reading";
my $line;
while($line = <CMDS>){
    chomp($line);
    # remove comments and leading and trailing space
    $line =~ s/\s*\#.*$//;
    $line =~ s/^\s+//;
    $line =~ s/\s+$//;
    next if not length($line);

    # uppercase only
    $line =~ tr/a-z/A-Z/;
    
    # break the list apart and stash it
    my @words = split(/\s/,$line);
        
    # key word to which the others resolve is $words[0], the first one
    for(@words) {
        $thesaurus->{$_} = $words[0];
    }
}
close CMDS;

# now lets see what results with rewriting
print "> ";
while(<>) {
    
    my @result = ();
    my @words = ();

    chomp;
    
    tr/a-z/A-Z/; # uppercase only

    last if /^\s*$/; # end if no input
    
    @words = split;
    for(@words){
        if(not defined $thesaurus->{$_}){
            # increment or create entry for new word
            if(defined $newWords->{$_}) { ++ $newWords->{$_}; }
            else { $newWords->{$_} = 1; }
            
            push(@result,"?$_?"); # flag it in output
        }else{
            next if $thesaurus->{$_} eq $ignore;
            push(@result,$thesaurus->{$_});
        }
    }
    print join(" ",@result),"\n";
    print "> ";
}

print "These words were not recognized:\n";
for (keys %$newWords) {
    print "$_\t\t$newWords->{$_}\n";
}
exit;

[download]

In reply to Re: NLP - natural language regex-collections? by perlcapt
in thread NLP - natural language regex-collections? by erix

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.