Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: Trying to set up a concordance using linguistic rules

by benizi (Hermit)
on Feb 28, 2008 at 05:13 UTC ( #670822=note: print w/replies, xml ) Need Help??


in reply to Re^2: Trying to set up a concordance using linguistic rules
in thread Trying to set up a concordance using linguistic rules

The fact that WordNet only contains open-class words might be a benefit, actually. If it's not in WordNet, the OP probably doesn't want it in the concordance.

...except that that idealizes WN's coverage of English. You'd probably do well, though with the added heuristic: "and it's a short, uncapitalized word". (i.e. "Short, uncapitalized words that aren't in WN should probably be ignored.")

...and it ignores the problem of homonyms. (e.g. 'in' has 7 senses in WordNet)

Nonetheless, felt the need to "defend" WN (since someone's quoting from the FAQ I wrote).

Better solutions have been mentioned elsewhere in the thread. But, I'll also add that my approach to the problem would probably be more Information Retrieval oriented. I'd use KStem, a stemming algorithm whose output is (in the usual case) an actual word. It tries to correct some of the problems with Porter. That paper is pretty academic (but good). There's also downloadable Java code.

Then I'd just use tf-idf style weights to pick out the interesting words.

Seems easier than parsing, even if it's shallow parsing.

  • Comment on Re: Trying to set up a concordance using linguistic rules

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://670822]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (4)
As of 2022-08-12 02:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?