Re^2: Trying to set up a concordance using linguistic rules

in reply to Re: Trying to set up a concordance using linguistic rules
in thread Trying to set up a concordance using linguistic rules

Given that WordNet explicitly excludes "determiners, prepositions, pronouns, conjunctions, and particles", I don't really see how a module providing an interface to this (otherwise excellent) resource might help the OP.

Comment on Re^2: Trying to set up a concordance using linguistic rules

Replies are listed 'Best First'.
Re: Trying to set up a concordance using linguistic rules by benizi (Hermit) on Feb 28, 2008 at 05:13 UTC
The fact that WordNet only contains open-class words might be a benefit, actually. If it's not in WordNet, the OP probably doesn't want it in the concordance. ...except that that idealizes WN's coverage of English. You'd probably do well, though with the added heuristic: "and it's a short, uncapitalized word". (i.e. "Short, uncapitalized words that aren't in WN should probably be ignored.") ...and it ignores the problem of homonyms. (e.g. 'in' has 7 senses in WordNet) Nonetheless, felt the need to "defend" WN (since someone's quoting from the FAQ I wrote). Better solutions have been mentioned elsewhere in the thread. But, I'll also add that my approach to the problem would probably be more Information Retrieval oriented. I'd use KStem, a stemming algorithm whose output is (in the usual case) an actual word. It tries to correct some of the problems with Porter. That paper is pretty academic (but good). There's also downloadable Java code. Then I'd just use tf-idf style weights to pick out the interesting words. Seems easier than parsing, even if it's shallow parsing.	[reply]

Replies are listed 'Best First'.

Re: Trying to set up a concordance using linguistic rules
by benizi (Hermit) on Feb 28, 2008 at 05:13 UTC

The fact that WordNet only contains open-class words might be a benefit, actually. If it's not in WordNet, the OP probably doesn't want it in the concordance.

...except that that idealizes WN's coverage of English. You'd probably do well, though with the added heuristic: "and it's a short, uncapitalized word". (i.e. "Short, uncapitalized words that aren't in WN should probably be ignored.")

...and it ignores the problem of homonyms. (e.g. 'in' has 7 senses in WordNet)

Nonetheless, felt the need to "defend" WN (since someone's quoting from the FAQ I wrote).

Better solutions have been mentioned elsewhere in the thread. But, I'll also add that my approach to the problem would probably be more Information Retrieval oriented. I'd use KStem, a stemming algorithm whose output is (in the usual case) an actual word. It tries to correct some of the problems with Porter. That paper is pretty academic (but good). There's also downloadable Java code.

Then I'd just use tf-idf style weights to pick out the interesting words.

Seems easier than parsing, even if it's shallow parsing.

[reply]

In Section Seekers of Perl Wisdom