in reply to Natural language processing

Misha --

Very interesting that you are looking for this. I am currently in the process of writing a set of tutorials on NLP with Perl, as I learn more about the subject myself. I have been working on it for a few days, and the first of the series is starting to reach a finished state. I'll send you a /msg when I complete it...

For now though, you might want to look up information about WordNet and the Link Grammar System on Google. There are Perl modules on CPAN for both of them. For WordNet, check out WordNet::QueryData, Lingua::Wordnet, WordNet::Similarity, and Lingua::Wordnet::Analysis. For the Link Grammar Parser, check out Lingua::LinkParser. There are many other modules - just browse around on CPAN in the Lingua:: group....

The field of natural language processing is huge, and it is difficult to find documentation out there that is accessible to the non-professional (one of my primary reasons for writing a tutorial). But the two tools I mentioned above are fairly easy to use, and if you can manage to get through the documentation, you should know enough at the end to know where to direct yourself from there. Browse around on Wikipedia as well to introduce yourself to some of the terminology used in the field. This will make reading the documentation much easier.

As far as the program you want to write -- training a computer program to pull useful information out of free conversation is not a simple thing to do. You'll see that after a few minutes of reading the Introduction To The Link Grammar Parser ;)

Here's a list of some of the interesting reading I've found lately. Hope this helps, and please let me know if you find anything interesting:



--mmmmtmmmm

Replies are listed 'Best First'.
Re^2: Natural language processing
by Gavin (Archbishop) on Sep 01, 2007 at 18:43 UTC
    You may also want to look at some of the Prolog modules.
[reply]
[reply]

        If you want to do Prolog programming in a Perl context, I definitely recommend Language::Prolog::Yaswi.

        It's an interface to the the very powerful and complete swi prolog system and allows access to Prolog from Perl and even defining Perl functions that can be used as predicates in Prolog.

        I once gave a lightning talk at the German Perl Workshop from which you can take some minimalistic code examples, which might help you over the entry barrier: yaswi.pdf. It's in german, but you can recognize the example code and probably reverse engineer the context.

        Integrating swi-prolog has some advantages over the use of AI::Prolog, mainly that it is one of the well known standard prolog systems which makes it possible to get high quality help from the Prolog community (eg. comp.lang.prolog. It comes with all bells and whistles and lots of additional libraries.

[reply]
Re^2: Natural language processing
by Gavin (Archbishop) on Sep 02, 2007 at 18:32 UTC
    AI::Prolog seems to offer all the advantages of the procedural programming language from Prolog and the flexibility of Perl.
    Constructing the various rules to cover all events which will be used to determine that a “deal has been done” will be difficult.

    In conversations the same sentences are not normally used again and again nor words in the same order, this will play havoc with predicates (rules and structures, new rules will need to be constantly added to cover changing speech patterns and vocabulary, therefore getting anything like 100% accuracy is not likely.

    But maybe that sort of accuracy is not required.

    Perhaps you could post a sample conversation so we can get an idea of what is involved.
[reply]

      A couple of quick notes. First, AI::Prolog is useful when there's simple pattern matching, but it's very slow. If you have a large search space, it's unusable. Also, I've always wanted to add support for DCDs (declarative clause grammars) which would make natural language processing much easier.

      Cheers,
      Ovid

      New address of my CGI Course.

[reply]

In Section Seekers of Perl Wisdom